Add Multiple Datasets

How to use the Add Multiple Datasets feature?

Context

When adding a large number of datasets, the process becomes tedious and repetitive.

With the Add multiple datasets feature, you can select multiple datasets from a source database at once. Through the multi-select form, you can control which source tables are to be ingested, and how and what data is ingested from those source tables. You can configure options like exclude/mask columns, flatten JSON, complete and incremental fetch, etc. from the Add multiple datasets form.

⚒️ Step-by-Step Process

This feature is live ⚡ for MySQL DB and Postgres DB sources in Data Imports

1. Create a Data Import of the above-mentioned source types. Establish a connection and move to the next stage of the process to Select Datasets for Ingestion.

2. There are two ways by which you can select the datasets (tables) from the source. Add Dataset - lets you select datasets one by one. Add multiple datasets - this enables you to add multiple datasets to the list in a single go. 🖱️Click on Add multiple datasets

   
3. Select the Database name from where the source tables are to be selected. On selecting the Database, Sprinkle fetches the source table details and prepares the form for you. In the example below we have selected the "classicmodels" database. The tabular form gets generated which shows all the tables in the database.

   

   Dataset Multiple Select form UI

4. 🖱️Check the Source tables that you want to be ingested into your warehouse. The checkbox on the top can be used to select all. Below we have selected all the tables from "classicmodels" database except the employee and office tables.

   

   Selected Source Tables: Highlighted Ingestion Mode
5. Make edits for the selected source tables to configure features like Exclude Columns, Mask Columns, and Mode while ingesting.

6. Change the Mode to Incremental if required. 📖 Read about the Ingestion modes supported in Sprinkle. On changing the mode to incremental, the detailed form opens up to enable you to configure the incremental ingestion settings.

7. Click on Create, once you are done with configuring the datasets. On successful addition of the dataset, you are directed to the listing page.

Other related features

Bulk Edit

The bulk edit is available on the Datasets listing page. The already selected dataset's configuration can be edited in a single go, just like in the case of adding multiple datasets.

⚒️How to use Bulk Edit?

1. Click on Bulk Edit

2. Select the Database - All the already selected datasets from the particular database with their related settings appear in a tabular form. Make necessary edits and click on Save.

   

🔄 Bulk Reset

Select the datasets on the listing page that you want to reset. The reset action deletes the warehouse table and storage data, but the table and storage data will be recreated upon the next ingestion schedule.

🗑️ Bulk Delete

Select the datasets on the listing page that you want to delete. The delete action deletes the warehouse table and storage data.