refactor(table): split the table hierarchy and materialize the store a table serves

docs/source/user_guide/format_table.rst

@@ -40,8 +40,9 @@ format table is an insert.
 
 Reading and writing
 -------------------
-A format table is not served through :cpp:func:`Catalog::GetTable`, which describes a managed
-table; use :cpp:func:`Catalog::GetFormatTable` instead.
+A format table is served through the same :cpp:func:`Catalog::GetTable` as every other table:
+its ``type`` option says what it is, and the call hands back the ``FormatTable`` it describes,
+through the ``Table`` interface the two kinds of table share.
 
 Reading and writing go through the entry points every other table uses: ``TableScan::Create``,
 ``TableRead::Create``, ``FileStoreWrite::Create`` and ``FileStoreCommit::Create``, each from its
@@ -58,8 +59,11 @@ a constructor taking one - which is what Java does through ``FormatTable.newRead
 
 .. code-block:: cpp
 
-   PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<paimon::FormatTable> table,
-                          catalog->GetFormatTable(paimon::Identifier("db", "tbl")));
+   PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<paimon::Table> loaded,
+                          catalog->GetTable(paimon::Identifier("db", "tbl")));
+   // The table's `type` option is `format-table`, so this is a `FormatTable`.
+   std::shared_ptr<paimon::FormatTable> table =
+       std::dynamic_pointer_cast<paimon::FormatTable>(loaded);
    paimon::ScanContextBuilder scan_builder(table);
    PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::ScanContext> scan_context,
                           scan_builder.Finish());
@@ -71,6 +75,12 @@ from one refuses ``SetTableSchema()``, ``WithFileSystem()``,
 ``WithFileSystemSchemeToIdentifierMap()``, ``WithCatalog()`` and a branch. Options
 given at the call still win over the ones the schema stored, as they do everywhere else.
 
+``FormatTable`` is a derived class of ``Table``, so the table serves the same calls itself: a
+caller holding one through a ``std::shared_ptr<paimon::Table>`` calls ``NewScan()``,
+``NewRead()``, ``NewWrite()`` or ``NewCommit()`` on it, and the operators come back for the table
+it is, whether or not the context carries one. This is the C++ side of what Java serves through
+one ``Table.newReadBuilder()`` for both kinds of table.
+
 A batch comes back in the table's column order, or the projection's when the read names one,
 behind the ``_VALUE_KIND`` field every ``BatchReader`` carries. Which columns a *file* is asked for
 and in what order is its own business: they are matched to the table's by name, as Java matches
@@ -89,12 +99,10 @@ outside the projection is never read, which is how Java scopes the same checks.
 named here only to describe what happens. A caller never needs them: a plan comes back as
 ``Plan``, a split as ``Split``, and a commit message as ``CommitMessage``.
 
-A catalog that can load a format table itself overrides ``Catalog::LoadFormatTable()``, the
-protected hook :cpp:func:`Catalog::GetFormatTable` calls. The file system catalog uses it to say
-its metadata lives under the table location, and the REST catalog to take the location and the
-schema from one response instead of two that could disagree. A catalog that does not override it
-still serves format tables, by reading the location and the schema through the virtuals every
-catalog has.
+Each catalog serves a format table from its own ``GetTable``. The file system catalog reads the
+schema under the table location, so it says the location carries the table's own metadata and a
+scan leaves the ``schema`` and ``branch`` directories below it alone; the REST catalog takes the
+location and the schema from one response, so the two cannot disagree.
 
 ``TableScan::ListPartitions()`` lists the partitions a scan can see. A format table answers it by
 listing directories; every other table type returns ``NotImplemented`` for now.

include/paimon/catalog/catalog.h

@@ -40,7 +40,6 @@ namespace paimon {
 using Instant = std::variant<std::string, int64_t>;
 
 class Database;
-class FormatTable;
 class Table;
 class View;
 class Schema;
@@ -148,6 +147,13 @@ class PAIMON_EXPORT Catalog {
 
     /// Gets a table.
     ///
+    /// The table's `type` option decides what is handed back: a `format-table` comes back as
+    /// the `FormatTable` it is - a directory of data files, with no snapshot and no manifest -
+    /// and every other implemented type as the managed table it is. A `T$system` identifier
+    /// names a read-only view instead, which comes back as the `SystemTable` it is. All go
+    /// through this one `Table` interface: `NewScan()` and `NewRead()` serve any of them,
+    /// `NewWrite()` and `NewCommit()` the writable ones.
+    ///
     /// @param identifier Identifier of the table to get.
     /// @return A result containing the table, or an error status.
     virtual Result<std::shared_ptr<Table>> GetTable(const Identifier& identifier) const = 0;
@@ -246,33 +252,6 @@ class PAIMON_EXPORT Catalog {
     /// @note Implementations returning true must also implement the internal
     /// `VersionManagedCatalog`.
     virtual bool SupportsVersionManagement() const;
-
-    /// Gets a format table: a directory of data files laid out like a standard Hive table.
-    ///
-    /// A format table carries no snapshots and no manifests, so it is loaded through its own
-    /// method rather than `GetTable()`. Reading and writing it go through the same `TableScan`,
-    /// `TableRead`, `FileStoreWrite` and `FileStoreCommit` entry points every other table uses:
-    /// hand the table back to `ScanContextBuilder`, `ReadContextBuilder`, `WriteContextBuilder` or
-    /// `CommitContextBuilder`, each of which takes one.
-    ///
-    /// @param identifier Identifier of the table to get.
-    /// @return A result containing the format table, or an error status if the table does not
-    /// exist or its `type` option is not `format-table`.
-    Result<std::shared_ptr<FormatTable>> GetFormatTable(const Identifier& identifier) const;
-
- protected:
-    /// Loads `identifier` as a format table, which is what `GetFormatTable()` hands back.
-    ///
-    /// The default reads the location and the schema through the virtuals above - two requests
-    /// that can disagree - and treats every directory below the location as table content. An
-    /// override answers both questions directly: whether location and schema come back in one
-    /// round trip, and whether this catalog keeps the table's metadata under the table path.
-    ///
-    /// @param identifier Identifier of the table to load.
-    /// @return A result containing the format table, or an error status if the table does not
-    /// exist or its `type` option is not `format-table`.
-    virtual Result<std::shared_ptr<FormatTable>> LoadFormatTable(
-        const Identifier& identifier) const;
 };
 
 }  // namespace paimon

include/paimon/catalog/table.h

@@ -33,50 +33,48 @@
 
 namespace paimon {
 
-class TableScan;
+class CommitContext;
+class FileStoreCommit;
+class FileStoreWrite;
+class ReadContext;
+class ScanContext;
 class TableRead;
+class TableScan;
+class WriteContext;
 
 /// A table provides basic abstraction for table type.
+///
+/// The state a table of every kind carries - file system, root path, schema and identifier - lives
+/// here, and each kind of table fills the operations in: the operators below create this table's
+/// scan, read, write or commit, from a context the caller finishes through its builder. What the
+/// context asks for that this table cannot honour is refused by name by the implementation rather
+/// than silently dropped.
 class PAIMON_EXPORT Table {
  public:
-    /// @param file_system File system the table's data is accessed through.
-    /// @param root_path Root path of the table in the file system.
-    /// @param uuid Metastore UUID, or empty if unavailable.
-    Table(const std::shared_ptr<FileSystem>& file_system, const std::string& root_path,
-          const std::shared_ptr<Schema>& schema, const Identifier& identifier,
-          const std::string& uuid = "");
-
-    ~Table() = default;
+    virtual ~Table() = default;
 
     /// Returns the file system the table was opened with, which serves the table's
     /// files. A catalog that hands out per-table file systems opens a table with the
-    /// one for that table; it is null when the opener had none.
+    /// one for that table; it is null when the opener had none. The `NewXxx`
+    /// operations read the table through it.
     std::shared_ptr<FileSystem> GetFileSystem() const {
         return file_system_;
     }
 
     /// Returns the root path of the table in its file system. It is empty when the
     /// opener had no location for the table, which a mock in a test is one case of.
+    /// The `NewXxx` operations serve the table from it.
     const std::string& GetRootPath() const {
         return root_path_;
     }
 
-    /// A name to identify this table.
-    std::string Name() const {
-        return identifier_.GetObjectName();
-    }
-
-    /// Full name of the table, default is database.tableName.
-    std::string FullName() const {
-        return identifier_.GetFullName();
-    }
+    /// A name to identify this table, which each kind of table answers for itself: a managed or
+    /// format table is named by its identifier, a system table by its own kind.
+    virtual std::string Name() const = 0;
 
-    /// Returns the branch the table was opened for, which its identifier carries; `main`
-    /// when it names none or a spelling of it, which its managers are bound to. A
-    /// malformed object name that hides the branch is reported.
-    Result<std::string> CurrentBranch() const {
-        return identifier_.GetBranchNameOrDefault();
-    }
+    /// Full name of the table, which each kind of table answers for itself. A managed or format
+    /// table answers database.tableName, a system table the same name as `Name()`.
+    virtual std::string FullName() const = 0;
 
     /// Returns the metastore UUID, falling back to `FullName()`.
     /// @note Use `CatalogUuid()` for commit requests to avoid sending a table name as an ID.
@@ -95,37 +93,60 @@ class PAIMON_EXPORT Table {
         return schema_;
     }
 
+    /// Copies this table with `dynamic_options` on top of the options it carries, which is
+    /// the precedence every context builder promises for a table it was handed rather than loaded
+    /// itself: an option given at the call wins over the one the schema stored.
+    ///
+    /// `type` is not overridable: it is structural and is read from the schema alone.
+    ///
+    /// @param dynamic_options Options given at the call.
+    /// @return A result containing the copied table, or the error copying it reports.
+    virtual Result<std::shared_ptr<Table>> Copy(
+        const std::map<std::string, std::string>& dynamic_options) const = 0;
+
     /// Creates a scan of this table from a finished scan context, the table-first entry point
-    /// that `TableScan::Create()` also builds through. The scan is derived from the context.
+    /// that `TableScan::Create()` loads the table the context names and hands the context to.
     /// @param context The finished scan context.
     /// @return The scan, or the error building it reports.
-    Result<std::unique_ptr<TableScan>> NewScan(std::unique_ptr<ScanContext> context) const;
+    virtual Result<std::unique_ptr<TableScan>> NewScan(
+        std::unique_ptr<ScanContext> context) const = 0;
 
     /// Creates a read of this table from a finished read context, the table-first entry point
-    /// that `TableRead::Create()` also builds through. The read is derived from the context.
+    /// that `TableRead::Create()` loads the table the context names and hands the context to.
     /// @param context The finished read context.
     /// @return The read, or the error building it reports.
-    Result<std::unique_ptr<TableRead>> NewRead(std::unique_ptr<ReadContext> context) const;
+    virtual Result<std::unique_ptr<TableRead>> NewRead(
+        std::unique_ptr<ReadContext> context) const = 0;
 
     /// Creates a write of this table from a finished write context, the table-first entry point
-    /// that `FileStoreWrite::Create()` also builds through. The write is derived from the context.
+    /// that `FileStoreWrite::Create()` loads the table the context names and hands the context to.
     /// @param context The finished write context.
     /// @return The write, or the error building it reports.
-    Result<std::unique_ptr<FileStoreWrite>> NewWrite(std::unique_ptr<WriteContext> context) const;
+    virtual Result<std::unique_ptr<FileStoreWrite>> NewWrite(
+        std::unique_ptr<WriteContext> context) const = 0;
 
     /// Creates a commit of this table from a finished commit context, the table-first entry point
-    /// that `FileStoreCommit::Create()` also builds through. The commit is derived from the
-    /// context.
+    /// that `FileStoreCommit::Create()` loads the table the context names and hands the context
+    /// to.
     /// @param context The finished commit context.
     /// @return The commit, or the error building it reports.
-    Result<std::unique_ptr<FileStoreCommit>> NewCommit(
-        std::unique_ptr<CommitContext> context) const;
+    virtual Result<std::unique_ptr<FileStoreCommit>> NewCommit(
+        std::unique_ptr<CommitContext> context) const = 0;
+
+ protected:
+    /// @param file_system File system the table's data is accessed through.
+    /// @param root_path Root path of the table in the file system.
+    /// @param uuid Metastore UUID, or empty if unavailable.
+    Table(const std::shared_ptr<FileSystem>& file_system, const std::string& root_path,
+          const std::shared_ptr<Schema>& schema, const Identifier& identifier,
+          const std::string& uuid = "");
 
- private:
     std::shared_ptr<FileSystem> file_system_;
     std::string root_path_;
     std::shared_ptr<Schema> schema_;
     Identifier identifier_;
+
+ private:
     std::string uuid_;
 };
 

include/paimon/commit_context.h

@@ -151,20 +151,23 @@ class PAIMON_EXPORT CommitContextBuilder {
     ///
     /// There is no commit user: a format table keeps no snapshot to record one in.
     ///
-    /// @param table The format table to commit to, as `Catalog::GetFormatTable()` hands it back.
+    /// @param table The format table to commit to, as `Catalog::GetTable()` hands it back.
     explicit CommitContextBuilder(const std::shared_ptr<FormatTable>& table);
 
     ~CommitContextBuilder();
 
-    /// Set a configuration options map to set some option entries which are not defined in the
-    /// table schema or whose values you want to overwrite.
+    /// Set a configuration options map. The options take part in loading the table a commit
+    /// serves: the load merges them over the ones the table's schema carries and the table it
+    /// hands back serves the merged ones. A caller holding a table already loaded - a load
+    /// that is over - changes it through `Table::Copy()`; an option contradicting a table held
+    /// that way is refused when a commit is asked of it. A format table takes the options as
+    /// its own.
     /// @note The options map will clear the options added by `AddOption()` before.
     /// @param options The configuration options map.
     /// @return Reference to this builder for method chaining.
     CommitContextBuilder& SetOptions(const std::map<std::string, std::string>& options);
 
-    /// Add a single configuration option which is not defined in the table schema or whose value
-    /// you want to overwrite.
+    /// Add a single configuration option, served the way `SetOptions()` describes.
     ///
     /// If you want to add multiple options, call `AddOption()` multiple times or use `SetOptions()`
     /// instead.

include/paimon/file_store_commit.h

@@ -50,7 +50,8 @@ class CommitMessage;
 /// manages versions. `UseRESTCatalogCommit()` only prepares a request for the caller to send.
 class PAIMON_EXPORT FileStoreCommit {
  public:
-    /// Create an instance of `FileStoreCommit`.
+    /// Create an instance of `FileStoreCommit`: the table the context names is loaded first, then
+    /// the context is handed to that table's `NewCommit`.
     ///
     /// @param context A unique pointer to the `CommitContext` used for commit operations.
     ///

include/paimon/file_store_write.h

@@ -42,7 +42,8 @@ class WriteContext;
 /// Interface for write operations in a file store.
 class PAIMON_EXPORT FileStoreWrite {
  public:
-    /// Create an instance of `FileStoreWrite`.
+    /// Create an instance of `FileStoreWrite`: the table the context names is loaded first, then
+    /// the context is handed to that table's `NewWrite`.
     ///
     /// @param context A unique pointer to the `WriteContext` used for write operations.
     ///

include/paimon/read_context.h

@@ -246,7 +246,7 @@ class PAIMON_EXPORT ReadContextBuilder {
     /// `SetTableSchema()`, `WithFileSystem()`, `WithFileSystemSchemeToIdentifierMap()` and a
     /// branch are refused here rather than ignored.
     ///
-    /// @param table The format table to read, as `Catalog::GetFormatTable()` hands it back.
+    /// @param table The format table to read, as `Catalog::GetTable()` hands it back.
     explicit ReadContextBuilder(const std::shared_ptr<FormatTable>& table);
 
     ~ReadContextBuilder();
@@ -324,15 +324,18 @@ class PAIMON_EXPORT ReadContextBuilder {
     ///       When set, read_field_ids and read_field_names are ignored.
     ReadContextBuilder& SetReadSchema(std::unique_ptr<ArrowSchema> read_schema);
 
-    /// Set a configuration options map to set some option entries which are not defined in the
-    /// table schema or whose values you want to overwrite.
+    /// Set a configuration options map. The options take part in loading the table a read
+    /// serves: the load merges them over the ones the table's schema carries and the table it
+    /// hands back serves the merged ones. A caller holding a table already loaded - a load
+    /// that is over - changes it through `Table::Copy()`; an option contradicting a table held
+    /// that way is refused when a read is asked of it. A format table takes the options as its
+    /// own.
     /// @note The options map will clear the options added by `AddOption()` before.
     /// @param options The configuration options map.
     /// @return Reference to this builder for method chaining.
     ReadContextBuilder& SetOptions(const std::map<std::string, std::string>& options);
 
-    /// Add a single configuration option which is not defined in the table schema or whose value
-    /// you want to overwrite.
+    /// Add a single configuration option, served the way `SetOptions()` describes.
     ///
     /// If you want to add multiple options, call `AddOption()` multiple times or use `SetOptions()`
     /// instead.