foundry_dev_tools.clients.catalog module

foundry_dev_tools.clients.catalog module#

Implementation of the foundry-catalog API.

class foundry_dev_tools.clients.catalog.CatalogClient[source]#

Bases: APIClient

To be implemented/transferred.

api_name: ClassVar[str] = 'foundry-catalog'#
list_dataset_files(dataset_rid, end_ref='master', page_size=1000, logical_path=None, page_start_logical_path=None, include_open_exclusive_transaction=False, exclude_hidden_files=False, temporary_credentials_auth_token=None)[source]#

Same as CatalogClient.api_get_dataset_view_files3(), but iterates through all pages.

Parameters:

  • dataset_rid (str) – the dataset rid

  • end_ref (str) – branch or transaction rid of the dataset

  • page_size (int) – the maximum page size returned

  • logical_path (str | None) – If logical_path is absent, returns all files in the view. If logical_path matches a file exactly, returns just that file. Otherwise, returns all files in the “directory” of logical_path: (a slash is added to the end of logicalPath if necessary and a prefix-match is performed)

  • page_start_logical_path (str | None) – if specified page starts at the given path, otherwise at the beginning of the file list

  • include_open_exclusive_transaction (bool) – if files added in open transaction should be returned as well in the response

  • exclude_hidden_files (bool) – if hidden files should be excluded (e.g. _log files)

  • temporary_credentials_auth_token (str | None) – to generate temporary credentials for presigned URLs

Returns:

[
    {
        "logicalPath": "..",
        "pageStartLogicalPath": "..",
        "includeOpenExclusiveTransaction": "..",
        "excludeHiddenFiles": "..",
    },
]

Return type:

list[FileResourcesPage]

api_get_dataset_view_files3(dataset_rid, end_ref, page_size, logical_path=None, page_start_logical_path=None, include_open_exclusive_transaction=False, exclude_hidden_files=False, start_transaction_rid=None, temporary_credentials_auth_token=None, **kwargs)[source]#

Returns files in the dataset view matching the specified parameters.

Parameters:

  • dataset_rid (api_types.DatasetRid) – the dataset rid

  • end_ref (api_types.View) – branch or transaction rid of the dataset

  • page_size (int) – the maximum page size returned

  • logical_path (api_types.PathInDataset | None) – If logical_path is absent, returns all files in the view. If logical_path matches a file exactly, returns just that file. Otherwise, returns all files in the “directory” of logical_path: (a slash is added to the end of logicalPath if necessary and a prefix-match is performed)

  • page_start_logical_path (api_types.PathInDataset | None) – if specified page starts at the given path, otherwise at the beginning of the file list

  • include_open_exclusive_transaction (bool) – if files added in open transaction should be returned as well in the response

  • exclude_hidden_files (bool) – if hidden files should be excluded (e.g. _log files)

  • start_transaction_rid (api_types.View | None) – start transaction rid

  • temporary_credentials_auth_token (str | None) – to generate temporary credentials for presigned URLs

  • **kwargs – gets passed to APIClient.api_request()

Returns:

the response contains a json dict with the following keys: values: an array of file resource objects nextPageToken: which can be used for the next request as start_transaction_rid

Return type:

response

api_get_events(types, limit=None, page_token=None, **kwargs)[source]#

Returns a page of events filtered by the types parameter.

Parameters:

  • types (set[str]) – types to filter for

  • limit (int | None) – limit the maximum numbers of events per page

  • page_token (str | None) – for pagination

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_create_dataset(dataset_path, **kwargs)[source]#

Creates dataset at specified path.

Parameters:

  • dataset_path (api_types.FoundryPath) – path on foundry for the to be created dataset

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_get_dataset(dataset_rid, **kwargs)[source]#

Returns rid and fileSystemId of dataset.

Parameters:

  • dataset_rid (api_types.DatasetRid) – the dataset rid

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_delete_dataset(dataset_rid, **kwargs)[source]#

Deletes the dataset.

Parameters:

  • dataset_rid (api_types.DatasetRid) – the dataset rid

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_set_transaction_type(dataset_rid, transaction_rid, transaction_type, **kwargs)[source]#

Set transaction type.

Parameters:

  • dataset_rid (api_types.DatasetRid) – dataset rid

  • transaction_rid (api_types.TransactionRid) – transaction rid

  • transaction_type (api_types.FoundryTransaction) – foundry transaction type, see api_types.FoundryTransaction

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_start_transaction(dataset_rid, branch_id, record=None, provenance=None, user_id=None, start_transaction_type=None, **kwargs)[source]#

Start a transaction on a dataset.

Parameters:

  • dataset_rid (api_types.DatasetRid) – dataset rid to start transaction on

  • branch_id (api_types.DatasetBranch) – the dataset branch

  • record (dict[str, Any] | None) – record

  • provenance (dict | None) – provenance for transaction

  • user_id (str | None) – start transaction as another user, needs foundry:set-user-id permissions

  • start_transaction_type (api_types.FoundryTransaction | None) – transaction type, default is APPEND

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_commit_transaction(dataset_rid, transaction_rid, record=None, provenance=None, do_sever_inherited_permissions=None, **kwargs)[source]#

Commit a transaction on a dataset.

Parameters:

  • dataset_rid (api_types.DatasetRid) – dataset rid to start transaction on

  • transaction_rid (api_types.TransactionRid) – the transaction to commit

  • record (dict[str, Any] | None) – record

  • provenance (dict | None) – provenance for transaction

  • do_sever_inherited_permissions (bool | None) – wether dependant conditions are removed on the transaction

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_abort_transaction(dataset_rid, transaction_rid, record=None, provenance=None, do_sever_inherited_permissions=None, **kwargs)[source]#

Abort a transaction on a dataset.

Parameters:

  • dataset_rid (api_types.DatasetRid) – dataset rid to start transaction on

  • transaction_rid (api_types.TransactionRid) – the transaction to commit

  • record (dict[str, Any] | None) – record

  • provenance (dict | None) – provenance for transaction

  • do_sever_inherited_permissions (bool | None) – wether dependant conditions are removed on the transaction

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_get_transaction(dataset_rid, ref, **kwargs)[source]#

Get the transaction for a given ref.

Parameters:

  • dataset_rid (api_types.DatasetRid)

  • ref (api_types.Ref)

Return type:

requests.Response

api_get_reverse_transactions2(dataset_rid, start_ref, page_size, end_transaction_rid=None, include_open_exclusive_transaction=False, allow_deleted_dataset=None, **kwargs)[source]#

Get reverse transactions.

Parameters:

  • dataset_rid (api_types.DatasetRid) – dataset rid to get transactions

  • start_ref (api_types.View) – at what ref to start listing

  • page_size (int) – response page entry size

  • end_transaction_rid (api_types.TransactionRid | None) – at what transaction to stop listing

  • include_open_exclusive_transaction (bool | None) – include open exclusive transaction

  • allow_deleted_dataset (bool | None) – respond even if dataset was deleted

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_create_branch(dataset_rid, branch_id, parent_ref=None, parent_branch_id=None, **kwargs)[source]#

Creates a branch on a dataset.

Parameters:

  • dataset_rid (api_types.DatasetRid) – the dataset

  • branch_id (api_types.DatasetBranch) – the branch to create

  • parent_ref (api_types.TransactionRid | None) – optionally the transaction off which the branch will be based

  • parent_branch_id (api_types.DatasetBranch | None) – optionally a parent branch name, otherwise a root branch

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_update_branch(dataset_rid, branch, parent_ref=None, **kwargs)[source]#

Updates the latest transaction of branch ‘branch’ to the latest transaction of branch ‘parent_branch’.

Parameters:

  • dataset_rid (api_types.DatasetRid) – Unique identifier of the dataset

  • branch (api_types.DatasetBranch) – The branch to update (e.g. master)

  • parent_ref (api_types.View | None) – the name of the branch to copy the last transaction from or a transaction rid

  • **kwargs – gets passed to APIClient.api_request()

Returns:

example below for the branch response

Return type:

dict

{
    "id": "..",
    "rid": "ri.foundry.main.branch...",
    "ancestorBranchIds": [],
    "creationTime": "",
    "transactionRid": "ri.foundry.main.transaction....",
}
api_get_branch(dataset_rid, branch, **kwargs)[source]#

Returns branch information.

Parameters:

  • dataset_rid (api_types.DatasetRid) – Unique identifier of the dataset

  • branch (api_types.DatasetBranch) – Branch name

  • **kwargs – gets passed to APIClient.api_request()

Returns:

with keys id (name) and rid (unique id) of the branch.

Return type:

dict

api_get_branches(dataset_rid, **kwargs)[source]#

Returns branch names of dataset.

Parameters:

  • dataset_rid (api_types.DatasetRid) – Unique identifier of the dataset

  • **kwargs – gets passed to APIClient.api_request()

Returns:

list of dataset branch names

Return type:

list[str]

api_remove_dataset_file(dataset_rid, transaction_id, logical_path, recursive=False, **kwargs)[source]#

Removes the given file from an open transaction.

If the logical path matches a file exactly then only that file will be removed, regardless of the value of recursive. If the logical path represents a directory, then all files prefixed with the logical path followed by ‘/’ will be removed when recursive is true and no files will be removed when recursive is false. If the given logical path does not match a file or directory then this call is ignored and does not throw an exception.

Parameters:

  • dataset_rid (api_types.DatasetRid) – Unique identifier of the dataset

  • transaction_id (api_types.TransactionRid) – transaction rid

  • logical_path (api_types.FoundryPath) – logical path in the backing filesystem

  • recursive (bool) – recurse into subdirectories

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_add_files_to_delete_transaction(dataset_rid, transaction_id, logical_paths, **kwargs)[source]#

Adds files in an open DELETE transaction.

Files added to DELETE transactions affect the dataset view by removing files from the view.

Parameters:

  • dataset_rid (api_types.DatasetRid) – Unique identifier of the dataset

  • transaction_id (api_types.TransactionRid) – transaction rid

  • logical_paths (list[api_types.PathInDataset]) – files in the dataset to delete

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_get_dataset_stats(dataset_rid, end_ref='master', **kwargs)[source]#

Returns response from foundry catalogue stats endpoint.

Parameters:

  • dataset_rid (api_types.DatasetRid) – the dataset rid

  • end_ref (api_types.View) – branch or transaction rid of the dataset

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

dict:

sizeInBytes, numFiles, hiddenFilesSizeInBytes, numHiddenFiles, numTransactions

Contents