@grouparoo / dbt

Grouparoo's dbt Plugin enables you to configure Grouparoo within your dbt project using a shared config.

Create a Grouparoo Project Within your dbt Project

To integrate Grouparoo within your existing dbt project, you'll want to follow our installation docs and make sure you initiate your Grouparoo project in the same directory as your dbt project.

Install the dbt Plugin

To work with the dbt Plugin, you must first install it in an existing Grouparoo project. You can do this using grouparoo config and choosing to add the new App. You can also install a plugin with the Grouparoo CLI and use the install command from our CLI: grouparoo install @grouparoo/dbt

Both methods of installation add the package to your package.json file as a dependency, and also drops the Plugin in the grouparoo.plugins section in that same file, which enables it.

// package.json

{
  // ...
  "dependencies": {
    "@grouparoo/dbt": "...",
    // ...
  },
  "grouparoo": {
    "plugins": [
      "@grouparoo/dbt",
      // ...
    ]
  }
}

Once the Plugin is installed, you'll be working primarily with the CLI's configuration commands to get everything set up.

• BigQuery
• Postgres
• Redshift
• Snowflake

Orchestrating Grouparoo with dbt

You can use an App Refresh Query to orchestrate your Grouparoo Project based on dbt runs. An App Refresh Query checks for new data in a specific place in your warehouse as often as every minute. If new data is found, it will trigger all Schedules associated with that App. To get started with App Refresh Query, you'll want to make sure you have a single place for Grouparoo to check for data.

Step 1: Set up a meta table

You can use dbt's on-run-end hook to generate a new table that is updated upon the completion of each dbt run.

At its simplest, you can do this by adding a macro to your project that generates a table that is updated after each run. Add the following line to your dbt_project.yml:

on-run-end: "CREATE TABLE IF NOT EXISTS dbt_meta (last_run_at TIMESTAMP); INSERT INTO dbt_meta (last_run_at) VALUES (NOW());"

You can also generate a more complex dbt_meta table using a macro as seen in our dbt example app project.

Step 2: Configure an App Refresh Query

Assuming you have already installed a Plugin and completed the basic configuration for your App, you can now set up your App Refresh Query.

Start by heading to the Refresh tab on your App and select "Add an App Refresh Query".

If you've set up a meta table in Step 1, you can enter the following as your query to check for any new runs with zero errors:

SELECT MAX(last_run_at) FROM dbt_meta

Otherwise, enter whatever query you would like Grouparoo to use to check for new data. You can read more about configuring an App Refresh Query in the App Configuration Docs. You can also set how frequently Grouparoo should run this query. By default, it will be run every minute.

Click "Update" to save your changes.

Next Steps

That's it! While Grouparoo is running, it will now run run your query periodically to check for new data and trigger any associated schedules.

You'll also want to make sure you configure the other items in our Configuration Guide to get the most out of your Grouparoo instance and begin sending data to your Destinations.