Table metadata sync

Syncing table metadata saves information about tables and columns to Codatum, where it becomes available in the catalog and elsewhere.

Automatic sync runs for connections where roughly 7 days have passed since the last sync check. If you need an immediate update, sync manually.

SQL execution results are returned regardless of the table metadata sync status. If results don't seem up to date, you might be seeing a cached result (see the cache section of SQL block).

Table identity

Synced tables are managed by the underlying object on the warehouse, not by connection. Within a workspace, the same underlying object is treated as a single table.

The elements that determine identity vary by warehouse.

Warehouse	Matching elements
BigQuery	Project / dataset / table
Snowflake	Account / database / schema / table
Databricks	Host and SQL warehouse / catalog / schema / table
Redshift	Account, region, cluster (or namespace) / database / schema / table

For example, in BigQuery, tables are the same if the project, dataset, and table match. Even from different connections, they appear as a single entry in the catalog.

When multiple connections sync the same table, Codatum handles it as follows.

• The catalog shows it as a single table. You can see the related connections on the table detail.
• Descriptions, tags, and other annotations are shared across the same table.
• You can view it if you have read permission on any related connection, and edit it if you have run permission on any related connection. See Catalog permissions for details.

Information collected

Codatum imports the following from your warehouse (some fields might be empty depending on the warehouse or permissions).

• Table structure: database name, schema name (dataset name for BigQuery), table name, table type
• Table description
• Row count
• Table size (bytes)
• Table creation and update timestamps
• Column name, data type, column description, and nullability

The Other tab on the table detail shows whatever additional items are available, such as definition, data location, view query, and partition information.

Differences by warehouse

• Redshift: Column descriptions aren't imported during sync. Table update timestamps aren't imported either.
• BigQuery and others: Warehouse-specific information such as partitioning and clustering might appear in Other.

When sync runs

• Automatic sync: Runs for connections where roughly 7 days have passed since the last sync check.
• Manual sync: Open the target connection under global nav > Workspace settings > Data management > Connections, then select Resync Immediately in Table sync settings.
• When saving sync targets: Selecting Save and sync while editing Sync targets triggers a sync after the targets are updated.

Sync targets

Choose the databases/schemas (datasets) to sync from Table sync settings on the connection detail. You can also set Sync targets when creating the connection.

• Auto add datasets (BigQuery) / Auto add schemas (other warehouses): Automatically adds newly created databases/schemas, or ones you've newly been granted access to, to the sync targets (on by default). This doesn't apply once the number of schemas/datasets in a database exceeds a certain threshold.
• You can select up to 200 schemas/datasets.

The demo connection's sync targets are fixed and can't be edited. See Demo connection for details.

Sync status

The on-screen labels map to the following status codes.

Label	Code
QUEUED	PENDING
PROCESSING	RUNNING
SUCCESS	SUCCESS
WARNING	WARNING
FAILED	ERROR

You can check the sync history from Sync history in Table sync settings on the connection detail.

Where it's used

Synced table metadata is used in the following, among others.

• Catalog
• Autocomplete in SQL blocks
• Column information in table overviews across various features

Permissions

• Viewing synced table metadata is limited to users with read permission on a connection associated with the table.
• For how Codatum handles tables synced by multiple connections, see Table identity.
• Resync Immediately and changing sync targets require run permission on the target connection.

See Connection permissions for details.

Limits

• Number of synced tables: Depends on your contract (check under global nav > Workspace settings > Usage).
• Number of schemas (datasets) you can select as sync targets: Up to 200.
• Number of tables that can be synced per schema: Up to 1,000.

See also Limits (fixed limits) and Limits that depend on your contract.

If a table can't be found

1. Check the sync status for errors.
2. Confirm the database and table are selected as a sync target.
3. Confirm the connection's credentials have permission to read the table.
4. If it's still not found, run Resync Immediately.