gh-129724: gh-156504: Make marshal output independent of reference counts (GH-159192)

The marshal writer decided whether an object can be referenced later
(FLAG_REF) from its reference count, so the output for the same value
depended on what else in the process held a reference to its parts, and
differed between the default and the free-threaded builds' view of the
same objects.  This is the remaining cause of gh-129724 and the cause of
gh-156504.

FLAG_REF is now set exactly on the objects which are written more than
once in the dump.  The writer still makes one pass: it records the
position of the type byte of each object it remembers and of each
TYPE_REF, and when the dump is complete it sets the flags and renumbers
the references.  A reference count of one is only used to skip the
hashtable.  Interned strings are no longer flagged unconditionally.

The reader is unchanged and the format version is not bumped: both the
old and the new output are loaded by old and new readers.

PyMarshal_WriteObjectToFile() now builds the data in memory and writes
it with one fwrite(), since the flags are only known at the end.
Nothing is written if the object cannot be marshalled.

Author: gpshead

Doc/c-api/marshal.rst

@@ -54,6 +54,10 @@ and :exc:`MemoryError` if memory allocation fails.
    Marshal a Python object, *value*, to *file*.
    *version* indicates the file format.
 
+   The marshalled representation of *value* is built in memory
+   and then written to *file*.
+   Nothing is written if *value* cannot be marshalled.
+
    This function can fail, in which case it sets the error indicator.
    Use :c:func:`PyErr_Occurred` to check for that.
 
@@ -68,6 +72,11 @@ and :exc:`MemoryError` if memory allocation fails.
       It can use :c:func:`PyMarshal_WriteObjectToString` and write the
       result itself, or validate what was written.
 
+   .. versionchanged:: next
+      Previously, the data was written to *file* as it was produced,
+      using a buffer of constant size,
+      and incomplete data was written if *value* could not be marshalled.
+
 .. c:function:: PyObject* PyMarshal_WriteObjectToString(PyObject *value, int version)
 
    Return a bytes object containing the marshalled representation of *value*.

Lib/test/test_capi/test_marshal.py

@@ -121,13 +121,39 @@ def test_write_object_to_file(self):
                     write_object_to_file(obj, filename, version)
                     data = read_file(filename)
                     self.assertEqual(marshal.loads(data), obj)
+                    self.assertEqual(data, marshal.dumps(obj, version))
+
+            # Larger than a stdio buffer, with references to objects
+            # written long before.
+            shared = (float('1.5'), b'bytes' * 10)
+            obj = [shared, 'long line ' * 10_000, shared, [shared] * 10_000]
+            write_object_to_file(obj, filename, version)
+            data = read_file(filename)
+            self.assertEqual(data, marshal.dumps(obj, version))
+            obj2 = marshal.loads(data)
+            self.assertEqual(obj2, obj)
+            if version >= 3:
+                self.assertIs(obj2[2], obj2[0])
+                self.assertIs(obj2[3][-1], obj2[0])
 
             with self.assertRaises(SystemError):
                 write_object_to_file(NULL, filename, version)
 
             with self.assertRaisesRegex(ValueError, 'cannot marshal object objects'):
                 write_object_to_file(UNMARSHALLABLE, filename, version)
 
+            # Nothing is written if the object cannot be marshalled,
+            # even if the error is only detected after more data than
+            # fits in a stdio buffer.
+            for obj in (UNMARSHALLABLE,
+                        [1, 'abc', UNMARSHALLABLE],
+                        [shared, 'long line ' * 10_000, shared,
+                         UNMARSHALLABLE]):
+                with self.assertRaisesRegex(ValueError,
+                                            'cannot marshal object objects'):
+                    write_object_to_file(obj, filename, version)
+                self.assertEqual(read_file(filename), b'')
+
     def test_read_short_from_file(self):
         # Test PyMarshal_ReadShortFromFile()
         read_short_from_file = _testcapi.pymarshal_read_short_from_file