Wrong Patch Generated

The following sample of code ought to work, but instead it raises an error:

from jsonpatch import apply_patch, make_patch

old = [
    {"x": ["a", {"y": ["b"]}], "z": "a"},
    {"x": ["c", {"d": ["d"]}], "z": "c"},
    {},
]

new = [
    {"x": ["c", {"y": ["d"]}], "z": "c"},
    {},
]

patch = make_patch(old, new)

assert apply_patch(old, patch) == new

Traceback (most recent call last):
  File "/venv/lib/python3.9/site-packages/jsonpointer.py", line 275, in walk
    return doc[part]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/temp.py", line 18, in <module>
    assert apply_patch(old, patch) == new
  File "/venv/lib/python3.9/site-packages/jsonpatch.py", line 151, in apply_patch
    return patch.apply(doc, in_place)
  File "/venv/lib/python3.9/site-packages/jsonpatch.py", line 669, in apply
    obj = operation.apply(obj)
  File "/venv/lib/python3.9/site-packages/jsonpatch.py", line 324, in apply
    subobj, part = self.pointer.to_last(obj)
  File "/venv/lib/python3.9/site-packages/jsonpointer.py", line 194, in to_last
    doc = self.walk(doc, part)
  File "/venv/lib/python3.9/site-packages/jsonpointer.py", line 278, in walk
    raise JsonPointerException("index '%s' is out of bounds" % (part, ))
jsonpointer.JsonPointerException: index '1' is out of bounds

The generated patch is:

[
    {"op": "remove", "path": "/0/x/0"},
    {"op": "replace", "path": "/0/x/1/y/0", "value": "d"},
    {"op": "replace", "path": "/0/z", "value": "c"},
    {"op": "remove", "path": "/1/x"},
    {"op": "move", "from": "/1/z", "path": "/0/x/0"},
    {"op": "remove", "path": "/2"},
]

Which is clearly the result of the error above.

It looks like swapping these two items fixes the patch:

{"op": "remove", "path": "/0/x/0"}
{"op": "replace", "path": "/0/x/1/y/0", "value": "d"}

So that the remove operation happens second:

{"op": "replace", "path": "/0/x/1/y/0", "value": "d"}
{"op": "remove", "path": "/0/x/0"}

Removing this section of code (which I assume is some optimization heuristic) fixes the assertion: https://github.com/stefankoegl/python-json-patch/blob/master/jsonpatch.py#L828-L851

For those who need to work around this until a patch is released, the following code will fix this issue (though probably with a hit to performance):

from jsonpatch import _ST_REMOVE
from jsonpatch import DiffBuilder as _DiffBuilder
from jsonpatch import JsonPatch as _JsonPatch
from jsonpatch import JsonPointer, RemoveOperation, _path_join


def apply_patch(doc, patch, in_place=False, pointer_cls=JsonPointer):
    if isinstance(patch, (str, bytes)):
        patch = JsonPatch.from_string(patch, pointer_cls=pointer_cls)
    else:
        patch = JsonPatch(patch, pointer_cls=pointer_cls)
    return patch.apply(doc, in_place)


def make_patch(src, dst, pointer_cls=JsonPointer):
    return JsonPatch.from_diff(src, dst, pointer_cls=pointer_cls)


class JsonPatch(_JsonPatch):
    @classmethod
    def from_diff(
        cls,
        src,
        dst,
        optimization=True,
        dumps=None,
        pointer_cls=JsonPointer,
    ):
        json_dumper = dumps or cls.json_dumper
        builder = DiffBuilder(src, dst, json_dumper, pointer_cls=pointer_cls)
        builder._compare_values("", None, src, dst)
        ops = list(builder.execute())
        return cls(ops, pointer_cls=pointer_cls)


class DiffBuilder(_DiffBuilder):
    def _item_removed(self, path, key, item):
        new_op = RemoveOperation(
            {
                "op": "remove",
                "path": _path_join(path, key),
            },
            pointer_cls=self.pointer_cls,
        )
        new_index = self.insert(new_op)
        self.store_index(item, new_index, _ST_REMOVE)

@stefankoegl

@stefankoegl any ideas on what's going wrong here? I've narrowed down the issue about as much as I can without having to put in a bunch more time to figure out the underlying issue.

@stefankoegl

@stefankoegl do you have any hints you could give me on how to fix this?

It turns out that my patch hasn't complete fixed this bug and I'm in desperate need of a solution.

Unfortunately, I don't have anything I can suggest. I haven't been actively working on the project for quite some time now, so I'm not up-to-speed at the moment.

Turns out my hacked fix still works. The new error I was experiencing was caused by something else.

While looking into this issue I hit a related case in the same area that fails silently instead of raising: make_patch produces a patch that applies without any error but yields the wrong document. On jsonpatch 1.33:

import jsonpatch

src = [None, True, {}]
dst = [[{"1": 1, "a": "s"}], {"": {"a": None, "1": 0}}, {"a": True}]

patch = jsonpatch.make_patch(src, dst)
# [{'op': 'replace', 'path': '/0', 'value': [{'1': 1, 'a': 's'}]},
#  {'op': 'add', 'path': '/1', 'value': {'': {'a': None, '1': 0}}},
#  {'op': 'move', 'from': '/1', 'path': '/2/a'}]

result = patch.apply(src)
# result   == [[{'1': 1, 'a': 's'}], True, {'a': {'': {'a': None, '1': 0}}}]
# expected == dst
assert result == dst   # fails, and no exception was raised

The generated move from /1 (right after the add at /1) lands in the wrong place, so the applied result differs from dst with no error surfaced. It appears to come from the same move-optimization path as the crash in this issue, just showing up as a wrong document rather than an exception.

Add property-based tests using Hypothesis (#194)

Add property-based tests using Hypothesis

Navigation Menu

Repository navigation

Wrong Patch Generated #138

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Wrong Patch Generated #138

Description

Activity

rmorshea commented on Dec 24, 2021

rmorshea commented on Dec 24, 2021

rmorshea commented on Jan 4, 2022

rmorshea commented on Jun 26, 2022

rmorshea commented on Aug 12, 2022

stefankoegl commented on Aug 12, 2022

rmorshea commented on Aug 13, 2022

ckarnell commented on Sep 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions