Azure EventHub
Guide to integrate your Azure EventHub to Sprinkle
Last updated
Guide to integrate your Azure EventHub to Sprinkle
Last updated
Before setting up the datasource, learn about datasource concepts
To learn about Connection, refer
Log into Sprinkle application
Navigate to Datasources -> Connections Tab -> New Connection ->
Select Azure Eventhub
Provide all the mandatory details
Name: Name to identify this connection
EventHub Connection String: Provide in the format -> DefaultEndpointsProtocol=https;AccountName=XXXX;AccountKey=XXXXXXXXXXXXX;EndpointSuffix=core.windows.net
Test Connection
Create
Navigate to Datasources -> Datasources Tab -> Add ->
Select Azure Eventhub
Provide the name -> Create
Connection Tab:
From the drop-down, select the name of connection created in STEP-2
Update
Stream Name (Required)
Automatic Schema (Required):
Yes: Schema is automatically discovered by Sprinkle (Recommended)
Flatten Level (Required): Select from One Level or Multi Level.In one level, flattening will not be applied on complex type. They will be stored as string. In multi level, flattening will be applied in complex level till they become simple type.
No: Schema needs to be provided
Warehouse Schema: Provide the row schema
Destination Schema (Required) : Data warehouse schema where the table will be ingested into
Destination Table name (Required) : It is the table name to be created on the warehouse. If not given, sprinkle will create like ds_<datasourcename>_<tablename>
Create
In the Ingestion Jobs tab:
Trigger the Job, using Run button
To schedule, enable Auto-Run. Change the frequency if needed
To learn about datasource, refer
Datasets Tab: To learn about Dataset, refer . Add Dataset for each table that you want to replicate, providing following details
Destination Create Table Clause: Provide additional clauses to warehouse-create table queries such as clustering, partitioning, and more, useful for optimizing DML statements. on how to use this field.