Create an AI Data Engineer

AI Data Engineers automate complex data tasks with the predictability of code. They autonomously plan, build, test, and generate production-ready notebooks to clean, standardize, and structure your data. These AI agents adapt to new data sources and formats, and can be scheduled or triggered on demand.

Here are the steps to create a new AI Data Engineer.

Create an AI Data Engineer

1. Go to AI Data Engineer and Select New AI Data Engineer.

1. Populate the first section of Create a new AI Data Engineer Fields.

   1. Enter AI Data Engineer Name

   2. Enter the Databricks Workspace URL

   3. Select the credential from the drop-down

      1. Ensure you have followed the steps in Databricks Credential creation

      2. If you don't see the credential in the drop-down, this means it has not been shared with Osmos.

   4. Select Validate

2. Populate the second section of Create a new AI Data Engineer Fields.

   1. Choose Git repo - Each Osmos task will operate against this Git folder in your Databricks Workspace.

      1. A Git repo is required, and this repo must be shared with the Databricks credentials for it to appear in the list.

      2. If you don't see the Git repo you expect, make sure you have shared the Git repo with the Service Principal.

   2. Home folder - Enter the email associated with your Databricks user folder.

   3. Select Cluster - Make sure your workspace has a running all-purpose cluster started via the Databricks UI or API.

      1. The Cluster must be shared with your Service Principal and with "can attach" permissions.

      2. Serverless and job clusters are not supported.

   4. Select Save.