Run the set checks in draft validate

CHANGELOG.md

@@ -11,7 +11,7 @@
 - `in2lambda draft field replace FIELD OLD NEW` changes the wording inside a field that is already written, for the faults only an edit can fix - a brace the OCR dropped out of some maths, which no range of the source says correctly. OLD has to occur in the field exactly once, or the command is refused saying how many times it occurs; `--regex` reads it as a regular expression and NEW as what to replace it with. The field is left quoting the lines it was taken from, at the layer that wrote it, but recorded as edited and by whoever replaced the wording, so the change can be shown against the source.
 - `in2lambda spec run SPEC` runs a YAML file of selectors over the frozen source: it says which blocks are questions, parts and solutions, which to ignore, what to strip off the front of each one, and which of the four filters lays the solutions out. It fills in the draft's fields with the markdown of the lines each was taken from, records the spec's name and hash in the log so a replay runs the same file, and reports every block it made nothing of. Running an edited spec over a draft it has already filled in is refused, as freezing a document that has changed is: `in2lambda source add --start-over` begins the draft again. Reading a spec needs pyyaml, which the `convert` extra now installs alongside panflute. See [the spec page](https://lambda-feedback.github.io/in2lambda/spec.html) for the selectors and layouts.
 - A field quoted out of a list item is now dedented as commonmark reads the item: the marker comes off the first line and as much of the same width off every line under it. So a question written `1.  ` no longer carries its number, a continuation line no longer arrives indented far enough to be rendered as a code block, and a spec's `strip` is left with what pandoc does not read as a marker. Values written by `in2lambda spec run`, `in2lambda draft question add`, `in2lambda draft part add` and `in2lambda draft question solution` change accordingly; the ranges behind them still name the same source lines.
-- `in2lambda validate` checks a draft over as a whole and writes what it finds into it as a `report`: source blocks in no field and not marked ignore, two fields taken from the same lines, gaps in the numbering of the questions or their parts, parts nothing answers, and fields holding nothing. Each finding names the field and the lines it is about, so it can be acted on without reading the draft. Finding something is not a failure and the command still exits 0; the report is replaced by the next run of the checks and dropped by the next command that changes the draft, since it describes the draft as it stood.
+- `in2lambda validate` checks a draft over as a whole and writes what it finds into it as a `report`: source blocks in no field and not marked ignore, two fields taken from the same lines, gaps in the numbering of the questions or their parts, parts nothing answers, and fields holding nothing. Each finding names the field and the lines it is about, so it can be acted on without reading the draft. Finding something is not a failure and the command still exits 0; the report is replaced by the next run of the checks and dropped by the next command that changes the draft, since it describes the draft as it stood. It also checks over the set the draft describes, as a converted document is checked at export - maths delimiters, what KaTeX will not render, images the export would not carry, and the compile Lambda Feedback's PDF generator does where pandoc and xelatex are installed, with a warning saying what to install where they are not - and reports each of those against the draft field the text is written in, so that `in2lambda build` refuses them as it refuses anything else in the report.
 - `in2lambda build` writes the draft in this directory out as a Lambda Feedback set: one question per `qN.text` field, holding the parts written for it and the worked solutions, with the images those fields refer to under `media/`, as `in2lambda convert` writes a set - a field naming an image that is not beside the draft is refused saying which file is missing, since the checks read the draft and not the folder it is in, and a question's own solution written beside a solution for every part it has becomes a part of its own holding just that solution, as `convert` pairs them up. It is refused unless `in2lambda validate` has been run since the draft last changed - every command that changes one drops its report - and found nothing, and the refusal prints what the report says so it can be acted on without opening the draft. `in2lambda render` writes each question as a PDF instead, compiled as Lambda Feedback's own PDF generator compiles it, which needs pandoc and xelatex; it is gated on nothing, since looking at a draft is how what the checks found gets fixed. Both take `-o/--out`, as `convert` does.
 - Importing `in2lambda.katex_convert` no longer writes a file called `log` into the working directory. What it has to say about a converted expression goes to the `in2lambda.katex_convert` logger, which is silent unless the application configures logging.
 - The Python API is unchanged: `in2lambda.main.runner` and everything under `in2lambda.api` take the same arguments and return the same objects.

in2lambda/draft/__init__.py

@@ -16,7 +16,7 @@
 from typing import Any
 
 import in2lambda.spec
-from in2lambda.draft.report import checks, overlapping, uncovered
+from in2lambda.draft.report import _order, checks, overlapping, uncovered
 from in2lambda.source import (
     DRAFT,
     SourceError,
@@ -318,9 +318,16 @@ def replay(directory: str = ".") -> None:
     for entry in draft["log"]:
         apply(rebuilt, markdown, entry, directory)
     # The one thing in a draft that no command wrote: the checks did, over the draft the
-    # commands left, so rebuilding it is running them again rather than copying it.
+    # commands left, so rebuilding it is running them again rather than copying it. What
+    # `in2lambda.validation` found over the set is carried across instead, since it
+    # depends on whether xelatex and Node are installed and the draft does not: rebuilt
+    # here it would come out shorter on a machine whose toolchain is not the one that
+    # validated, and an untouched draft would be accused of having been edited.
     if "report" in draft:
-        rebuilt["report"] = checks(rebuilt)
+        carried = [
+            finding for finding in draft["report"] if finding["check"] == "problem"
+        ]
+        rebuilt["report"] = sorted(checks(rebuilt) + carried, key=_order)
 
     path = Path(directory) / DRAFT
     if serialise(rebuilt) != path.read_bytes():

in2lambda/draft/export.py

@@ -23,7 +23,7 @@
 from in2lambda.api.set import Set
 from in2lambda.json_convert.json_convert import _question_stem, _question_title
 from in2lambda.source import DRAFT, ConversionToolsMissing, SourceError, frozen
-from in2lambda.validation import _IMAGE, pdf
+from in2lambda.validation import _IMAGE, _location, pdf
 
 _QUESTION = re.compile(r"q(\d+)\.text")
 """A question's text, and the number that orders it."""
@@ -119,6 +119,66 @@ def as_set(draft: dict[str, Any], directory: str = ".") -> Set:
     return question_set
 
 
+def located(draft: dict[str, Any]) -> dict[str, str]:
+    """Which field of a draft each place `in2lambda.validation` reports against is.
+
+    :func:`as_set` read backwards. The validator names a question, a part and a field
+    of the export, which is no address in the draft that wrote it, so this walks the
+    fields the way :func:`as_set` walks them and must be changed with it.
+
+    Args:
+        draft: A draft, as `in2lambda.source.frozen` reads one.
+
+    Returns:
+        The draft's field key for each location of the set it describes, the question's
+        own location included - where a problem about the whole question, such as an
+        image the export would not contain, is reported. A part answered by its
+        question's solution is located at that solution, since that is the field to go
+        and edit. Places no field of the draft wrote - a part's answer, the empty part
+        a question written without any exports as - are not here: nothing is in them
+        for the validator to find.
+
+    Examples:
+        >>> from in2lambda.draft.export import located
+        >>> fields = {"q1.text": {"value": "State it."}, "q1.solution": {"value": "$x$"}}
+        >>> located({"fields": fields})
+        {'Question 1 ""': 'q1.text', 'Question 1 "", main text': 'q1.text', 'Question 1 "", part (a), worked solution': 'q1.solution'}
+        >>> fields["q1.p1.text"] = {"value": "Do it."}
+        >>> fields["q1.p1.solution"] = {"value": ""}
+        >>> located({"fields": fields})['Question 1 "", part (a), worked solution']
+        'q1.solution'
+    """
+    fields = draft["fields"]
+    where = {}
+    for number in sorted(
+        int(found[1]) for key in fields if (found := _QUESTION.fullmatch(key))
+    ):
+        where[_location(number, "")] = f"q{number}.text"
+        where[_location(number, "", field="main text")] = f"q{number}.text"
+        parts = sorted(
+            int(found[2])
+            for key in fields
+            if (found := _PART.fullmatch(key)) and int(found[1]) == number
+        )
+        solution = f"q{number}.solution"
+        for index, part in enumerate(parts):
+            where[_location(number, "", index, "text")] = f"q{number}.p{part}.text"
+            written = f"q{number}.p{part}.solution"
+            # On the value and not the key, as `as_set` decides it: a solution field
+            # written empty leaves the part for its question's solution to answer.
+            if fields.get(written, {}).get("value"):
+                where[_location(number, "", index, "worked solution")] = written
+            elif solution in fields:
+                where[_location(number, "", index, "worked solution")] = solution
+        if solution in fields and all(
+            fields.get(f"q{number}.p{part}.solution", {}).get("value") for part in parts
+        ):
+            # The part `as_set` appends for a question's solution with no part left for
+            # it to answer, which is the last one and holds nothing else.
+            where[_location(number, "", len(parts), "worked solution")] = solution
+    return where
+
+
 def build(directory: str = ".", output_dir: str = "out") -> Path:
     """Writes the draft in a directory out as a Lambda Feedback set, if it is clean.
 

in2lambda/draft/report.py

@@ -8,21 +8,27 @@
 to find out what is left to do, without reading the draft itself.
 
 Everything here reports, never refuses: what the checks found may well be deliberate, and
-deciding that is whoever is writing the draft's to do. Only what is in the draft is
-looked at - its blocks, its field keys, their ranges and their values - because what the
-text of a question says is `in2lambda.validation`'s, at export.
+deciding that is whoever is writing the draft's to do. The checks themselves read only
+what is in the draft - its blocks, its field keys, their ranges and their values - and
+what the text of a question says is `in2lambda.validation`'s: the set the draft describes
+is exported and checked over as well, so that maths Lambda Feedback will not render is
+reported against the field it is written in rather than found after uploading.
 """
 
 import re
+import warnings
 from pathlib import Path
 from typing import Any
 
+from in2lambda.draft.export import as_set, located
 from in2lambda.source import DRAFT, frozen, save
+from in2lambda.validation import pdf
 
 Finding = dict[str, Any]
 """One thing a check found: ``{"check", "field", "ranges", "message"}``.
 
-``check`` is which check found it, ``field`` the block id or field key it is about,
+``check`` is which check found it - ``problem`` where it was `in2lambda.validation`,
+over the set the draft describes - ``field`` the block id or field key it is about,
 ``ranges`` the lines in question as ``[[start, end], ...]``, and ``message`` a sentence
 naming all of that, so that a line of the report can be acted on by itself.
 """
@@ -236,15 +242,90 @@ def checks(draft: dict[str, Any]) -> list[Finding]:
         + _without_solutions(draft)
         + _empty(draft)
     )
-    return sorted(
-        found,
-        key=lambda finding: (
-            finding["ranges"][0][0] if finding["ranges"] else _UNPLACED,
-            finding["field"],
-        ),
+    return sorted(found, key=_order)
+
+
+def _order(finding: Finding) -> tuple[int | float, str]:
+    """Where a finding goes in a report: earliest line first, then by what it is about."""
+    return (
+        finding["ranges"][0][0] if finding["ranges"] else _UNPLACED,
+        finding["field"],
     )
 
 
+def problems(draft: dict[str, Any], directory: str = ".") -> list[Finding]:
+    """What `in2lambda.validation` finds in the set the draft describes.
+
+    The draft is exported as it stands and the set checked over - maths delimiters,
+    what KaTeX will not render, images the export would not carry, and the compile
+    Lambda Feedback's PDF generator does - so that a question that will not render is
+    reported while the draft is being written rather than after it is uploaded.
+
+    Args:
+        draft: A draft, as `in2lambda.source.frozen` reads one.
+        directory: Where the draft is, and so what the images it names are beside.
+
+    Returns:
+        One :data:`Finding` per problem, named by the field of the draft it is in
+        rather than by the question and part of the export, so that a line of it can be
+        acted on with `field replace`. A problem about no one field - the set as a
+        whole failing to compile - keeps the validator's own naming of where it is.
+
+    Warns:
+        UserWarning: pandoc or xelatex is not installed, so the set was not compiled.
+    """
+    where = located(draft)
+    if not where:
+        # A draft with no question in it yet describes an empty set, which has nothing
+        # to find and is not worth a xelatex run to find it in.
+        return []
+
+    missing = pdf.missing_tools()
+    if missing:
+        # As `_katex_rejections` does without Node: a check that cannot be run here says
+        # what to install and leaves the rest of the report alone.
+        warnings.warn(
+            "The set the draft describes was not compiled as the PDF generator would: "
+            "install " + " and ".join(missing),
+            stacklevel=2,
+        )
+
+    fields = draft["fields"]
+    found = []
+    for problem in as_set(draft, directory).problems(compile=not missing):
+        location = max(
+            (named for named in where if problem.location.startswith(named)),
+            key=len,
+            default="",
+        )
+        if location:
+            key = where[location]
+            ranges = fields[key]["ranges"]
+            # Whatever the location says past the field: KaTeX names the characters of
+            # it that it stopped at, and those are the field's characters here as well.
+            rest = problem.location[len(location) :]
+            finding = {
+                "check": "problem",
+                "field": key,
+                "ranges": ranges,
+                "message": f"{key}{_where(ranges)}{rest}: {problem.message}",
+            }
+        else:
+            finding = {
+                "check": "problem",
+                "field": "",
+                "ranges": [],
+                "message": str(problem),
+            }
+        if finding not in found:
+            # A question's solution answers every part of it that has no solution of its
+            # own, so one fault in it is found once per part. They are the same field,
+            # the same lines and the same wording: a second line of the report saying so
+            # is a `field replace` that would be refused for finding nothing to replace.
+            found.append(finding)
+    return found
+
+
 def validate(directory: str = ".") -> list[Finding]:
     """Checks the draft in a directory over and writes the report into it.
 
@@ -256,16 +337,20 @@ def validate(directory: str = ".") -> list[Finding]:
         directory: Where the ``draft.json`` to check is.
 
     Returns:
-        What the checks found, as it was written into the draft.
+        What the checks and `in2lambda.validation` found, as it was written into the
+        draft.
 
     Raises:
         DraftMissing: there is no draft in that directory.
         DraftUnreadable: what is there is not a draft anything here wrote.
         SourceUnreadable: the markdown the draft names has moved, or is not text.
         DraftExists: the markdown has changed since the draft was written from it, so
             the lines the report named would not be the lines it was written about.
+
+    Warns:
+        UserWarning: a check could not be run here - see :func:`problems`.
     """
     draft, _ = frozen(directory)
-    draft["report"] = checks(draft)
+    draft["report"] = sorted(checks(draft) + problems(draft, directory), key=_order)
     save(Path(directory) / DRAFT, draft)
     return draft["report"]

in2lambda/main.py

@@ -424,8 +424,12 @@ def validate() -> None:
 
     Reports source blocks in no field and not marked ignore, two fields taken from the
     same lines, gaps in the numbering of the questions or their parts, parts nothing
-    answers, and fields holding nothing. Finding something is not a failure: the report
-    is written into draft.json either way, and replaced by the next one.
+    answers, and fields holding nothing. The set the draft describes is checked over as
+    well - maths delimiters, what KaTeX will not render, images the export would not
+    carry, and the compile Lambda Feedback's PDF generator does where pandoc and xelatex
+    are installed - each against the field it is written in. Finding something is not a
+    failure: the report is written into draft.json either way, and replaced by the next
+    one.
     """
     with _message_not_traceback():
         report = in2lambda.draft.report.validate()