foundry_dev_tools.clients.magritte_coordinator module

Contents

foundry_dev_tools.clients.magritte_coordinator module#

Implementation of the foundry-stats API.

class foundry_dev_tools.clients.magritte_coordinator.MagritteCoordinatorClient[source]#

Bases: APIClient

The mothership that controls remote Magritte agents and cloud runtimes.

api_name: ClassVar[str] = 'magritte-coordinator'#
api_get_oidc_metadata(**kwargs)[source]#

Returns the OIDC Metadata of the Foundry Stack.

Parameters:

**kwargs – gets passed to APIClient.api_request()

Returns:

{‘issuer’: ‘https://pltroidcpublic<id>.s3.<region>.amazonaws.com/foundry’}

Return type:

dict

get_oidc_issuer()[source]#

Returns the OIDC issuer of the Foundry Stack.

Return type:

str

api_add_source_v3(config, description, runtime_platform_request, parent_rid, **kwargs)[source]#

Low level method to add a new source to Data Connection.

Use a high level method like MagritteCoordinatorClient.create_s3_direct_source() for easier usage.

Parameters:

  • config (dict) – Source Config, individual format for each supported Source.

  • description (dict) – Textual description field.

  • runtime_platform_request (dict) – For Direct Connect { “cloud”: {“networkEgresses”: []}, “type”: “cloud” }

  • parent_rid (FolderRid) – Rid of the parent Compass folder.

  • **kwargs – gets passed to APIClient.api_request()

Return type:

requests.Response

api_update_source_v2(source_rid, updated_source_config=None, patch_cloud_runtime_platform=None, **kwargs)[source]#

Updates a Source.

Parameters:

  • source_rid (Rid) – The Rid of the Magritte Source

  • updated_source_config (dict | None) – example {“type”:”s3-direct”,”url”:”s3://bucket1”}

  • patch_cloud_runtime_platform (dict | None) –

    example {

    ”secretsToUpdate”: {}, “removedSecrets”: [], “patchCloudIdentity”: {}, “patchOidcRuntime”: {},

    }

  • **kwargs – gets passed to APIClient.api_request()

Returns:

dict with top level key updatedSource {

”updatedSource”: {
“sourceConfig”: {

“type”: “s3-direct”, “url”: “s3://bucket”

}, “connections”: {}, “additionalSecrets”: {}, “mavenCoordinates”: {

”userProvided”: [], “foundryProvided”: []

}

}

}

Return type:

requests.Response

create_s3_direct_source(name, parent_rid, url, description='', region=None, catalog=None, sts_role_configuration=None, network_egress_policies=None, **source_kwargs)[source]#

Create a new Source of type s3-direct. Only supports Direct Connect sources.

Parameters:

  • name (str) – Name of the Source in Compass

  • parent_rid (FolderRid) – Rid of the parent Compass folder.

  • url (str) – S3 Url, e.g. s3://bucket-name

  • description (str | None) – Textual description field.

  • region (str | None) – AWS Region of the bucket.

  • catalog (dict | None) – Catalog Integration for Iceberg Tables.

  • sts_role_configuration (dict | None) – sts role configuration for Assuming a role to connect to the bucket.

  • network_egress_policies (set[Rid] | None) – Network Egress Policies required to connect to the bucket.

  • source_kwargs – Any other arguments get added to the “source” key of the payload.

Return type:

str

get_source_config(source_rid)[source]#

Returns the configuration of a Source.

Parameters:

source_rid (SourceRid)

Return type:

dict

api_bulk_get_source_config(source_rids, **kwargs)[source]#

Returns the configuration of a multiple sources.

Parameters:

source_rids (set[SourceRid])

Return type:

requests.Response

get_source_description(source_rid)[source]#

Returns the description of a Source.

Returns:

{‘apiName’: None, ‘description’: ‘the desc.’, ‘name’: ‘source-name’, ‘type’: ‘s3-direct’}

Return type:

dictionary with example payload

Parameters:

source_rid (SourceRid)

api_bulk_get_source_description(source_rids, **kwargs)[source]#

Returns the description of multiple Sources.

Parameters:

source_rids (set[SourceRid])

Return type:

requests.Response

api_get_runtime_platform(source_rid, **kwargs)[source]#

Get runtime platform information for a source.

Returns:

example format {

“type”: “cloud”, “cloud”: {

”credential”: {

“credentialId”: “ri.credential..credential.7ddaaa49-1337-1337-a0c0-8b621fa30364”, “secretNames”: []

}, “networkEgressPolicies”: {

”networkEgressPolicies”: []

}, “cloudIdentity”: null, “oidcRuntime”: null

}

}

Return type:

response

Parameters:

source_rid (SourceRid)

set_network_egress_policies(source_rid, network_egress_policies)[source]#

Sets the network egress policies for a source, overwrites all pre-existing egress policies.

Parameters:

  • source_rid (SourceRid)

  • network_egress_policies (set[NetworkEgressPolicyRid])

get_network_egress_policies(source_rid)[source]#

Returns list of network egress policies for a source.

Parameters:

source_rid (SourceRid)

Return type:

list[NetworkEgressPolicyRid]

add_network_egress_policy(source_rid, network_egress_policy_rid)[source]#

Adds a single egress policy to a source. Additive operation.

Parameters:

  • source_rid (SourceRid)

  • network_egress_policy_rid (NetworkEgressPolicyRid)

delete_network_egress_policy(source_rid, network_egress_policy_rid)[source]#

Deletes a single egress policy from a source.

Parameters:

  • source_rid (SourceRid)

  • network_egress_policy_rid (NetworkEgressPolicyRid)

enable_s3_oidc_runtime(source_rid)[source]#

Enables OpenID Connect (OIDC) in (S3) Source.

Parameters:

source_rid (SourceRid)

disable_s3_oidc_runtime(source_rid)[source]#

Disables OpenID Connect (OIDC) in (S3) Source.

Parameters:

source_rid (SourceRid)

api_modify_source_usage_restrictions(source_rid, payload, **kwargs)[source]#

Update usage restrictions for a source.

Parameters:

  • source_rid (SourceRid) – the source to modify usage restrictions for

  • payload (dict) – the payload to modify the usage restrictions for, example: {“toRestrict”: [], “toEnable”: [{“type”: “stemmaRepository”, “stemmaRepository”: {}}]}

  • kwargs – any additional keyword arguments are passed to the requests library.

Return type:

requests.Response

enable_code_imports(source_rid, to_enable)[source]#

Enable code import on Magritte Source.

Parameters:

  • source_rid (SourceRid) – Magritte Source Rid.

  • to_enable (list[CodeResourceType]) – Pick from “stemmaRepository”, “computeModule”, “eddiePipeline”

restrict_code_imports(source_rid, to_restrict)[source]#

Restrict code import on Magritte Source.

Parameters:

  • source_rid (SourceRid) – Magritte Source Rid.

  • to_restrict (list[CodeResourceType]) – Pick from “stemmaRepository”, “computeModule”, “eddiePipeline”

api_bulk_get_usage_restrictions_for_source(source_rids, **kwargs)[source]#

Get information about which types of code resources allow the import of the requested sources.

Parameters:

source_rids (list[SourceRid])

Return type:

requests.Response

api_get_export_state_for_source(source_rid, **kwargs)[source]#

Returns the current export config of a source.

Returns:

example format {

“isEnabled”: True, “exportableMarkings”: [“marking_id”], “isEnabledWithoutMarkingsValidation”: False

}

Parameters:

source_rid (SourceRid)

Return type:

requests.Response

api_update_export_state_for_source(source_rid, is_enabled, is_enabled_without_markings_validation=None, **kwargs)[source]#

Updates Export config for a source.

Parameters:

  • source_rid (SourceRid) – Magritte Source Rid.

  • is_enabled (bool) – Whether the export is allowed or not.

  • is_enabled_without_markings_validation (bool | None) – Whether export without marking validation is allowed or not.

  • kwargs – any additional keyword arguments are passed to the requests library.

Return type:

requests.Response

api_add_exportable_markings_for_source(source_rid, exportable_markings, **kwargs)[source]#

Adds exportable markings for a source.

Parameters:

  • source_rid (SourceRid)

  • exportable_markings (list[MarkingId])

Return type:

requests.Response

api_remove_exportable_markings_for_source(source_rid, markings_to_remove, **kwargs)[source]#

Remove exportable markings for a source.

Parameters:

  • source_rid (SourceRid)

  • markings_to_remove (list[MarkingId])

Return type:

requests.Response

create_snowflake_source(name, parent_rid, account_identifier, description='', role=None, db=None, schema=None, warehouse=None, network_egress_policies=None)[source]#

Create a new Source of type snowflake. Only supports Direct Connect sources.

Parameters:

  • name (str) – Name of the Source in Compass

  • parent_rid (FolderRid) – Rid of the parent Compass folder.

  • account_identifier (str) – https://docs.snowflake.com/en/user-guide/admin-account-identifier.html

  • description (str | None) – Textual description field.

  • role (str | None) – The default access control role to use in the Snowflake session initiated by the driver

  • db (str | None) – The default database to use once connected

  • schema (str | None) – The default schema to use once connected

  • warehouse (str | None) – The virtual warehouse to use once connected

  • network_egress_policies (set[Rid] | None) – Network Egress Policies required to connect to the Snowflake Account. Front-door plus internal stage bucket is usually required.

Return type:

str

enable_snowflake_external_oauth(source_rid)[source]#

Enables external OAuth (OIDC) in (Snowflake) Source.

Parameters:

source_rid (SourceRid)

disable_snowflake_external_oauth(source_rid)[source]#

Disables external OAuth (OIDC) in (Snowflake) Source.

Parameters:

source_rid (SourceRid)

Contents