Sprinkle Docs
  • What is Sprinkle?
  • Quick Start
  • Analysing your data
    • 🔭Analytics Overview
    • 💠Data Models
      • *️Variables
      • 🌲Hierarchies
      • 🤿Column Mask
    • 🎉Switch to New Reports & Dashboards
    • 🆕Reports
      • Overview
      • Build Using Tables
        • Create a new Report
        • Layout and options
        • Build and Format - Overview
        • Apply Row Limits
        • Identify Date Columns
        • Filter your data
        • Visualizations
          • Table
          • Pivot
          • Line Chart
          • Bar Chart
          • Column Chart
          • Area Chart
          • Combo Chart
          • Scatter & Bubble Plot
          • Pie Chart
          • Funnel Chart
          • Stat Card
          • Point Map
          • Heat Map
          • Radial gauge chart
        • Advanced Features
          • Custom Analysis
          • Variables
          • Table & Quick Calculations
          • Drill - Hierarchical & Date
          • Break Out
          • RLS in Table reports
          • Scheduled Exports
          • Embedding Table Reports
      • Build Using Models
        • Create a new report
        • Layout and options
        • Visualizations
        • Advanced Features
      • Build SQL Reports
        • Create a new Report
        • Layout and options
        • Writing a SQL Code on Editor
        • Visualizations
        • Variables in SQL Reports
    • 🆕Dashboards
      • 🌀Filters
      • 👆Click Behaviour
      • ⏰Data Alerts
      • 🗓️Date Drill
      • 📤Scheduled Exports
      • 🔗Embed link
      • 🖥️Dashboard layout
      • 📱Mobile Dashboards
  • Transforming your data
    • 🔰SQL Transform
    • 📓Python Notebooks
  • Integrating your data
    • ☁️Destination Warehouses
      • AWS Athena
        • Manage storage of Flow tables
      • AWS Redshift
      • Azure Synapse
      • Databricks
      • Google BigQuery
      • MySQL
      • Postgres
      • Snowflake
      • SQL Server
      • K8 Setup
        • AWS EKS
        • Google GKE
        • Azure AKS
    • ⚙️Warehouse & Storage Setup
  • Ingesting your data
    • ☄️Data Imports
      • Databases
        • Azure Cosmos DB
        • Azure Table Storage
        • Google BigQuery
        • Mongo DB
        • MySQL DB
        • Oracle DB
        • Postgres DB
        • SQL Server DB
        • Features
          • Ingestion Modes
          • Add Multiple Datasets
          • CDC Setup
            • CDC setup in Mysql
            • CDC setup in Postgres
            • CDC setup in Mongo
            • CDC setup in SQL Server
          • Destination Create Table Clause
          • SSH Tunnel Setup
      • Files
        • AWS S3
        • AWS S3 External
        • Azure Blob
        • FTP
        • Google Cloud Storage
        • Google Sheet
        • SFTP
      • Applications
        • Apple Search Ads
        • Appsflyer
        • Branch
        • Clevertap
        • Facebook Ads
        • Freshdesk
        • Freshsales
        • Google Ads
        • Google Ads V2
        • Google Analytics
        • Google Analytics 4
        • Google Analytics MCF
        • Google Search Console
        • Hubspot
        • Impact Ads
        • Intercom
        • Klaviyo
        • Leadsquared
        • LinkedIn Ads
        • Magento
        • Mailchimp
        • Marketo
        • Mixpanel
        • MoEngage
        • Rocketlane
        • Salesforce
        • SAP S4
        • Shopify
        • Snapchat Marketing
        • TikTok Ads
        • WooCommerce
        • Zendesk Chat
        • Zendesk Support
        • Zoho Analytics
        • Zoho Books
        • Zoho CRM
        • Zoho Desk
        • Zoho Invoice
        • Zoho Subscription
      • Events
        • Apache Kafka
        • AWS Kinesis
        • Azure EventHub
    • 📤File Uploads
    • 🤖API Pulls
    • 🕸️Webhooks
  • Collaborating on data
    • 📤Sharing
    • 💬Comments
    • ⚡Activity
    • 🏷️Labels
  • Managing Schedules and Data Refreshes
    • ⏱️Schedules
    • 🔔Notifications
  • User Management
    • 🔑Access Management
    • 🧑‍🤝‍🧑Groups
    • 📂Folders
    • 🔄Syncing users, groups and RLS
    • 📧Azure AD Integration
  • Data Security & Privacy
    • 🔐Security at Sprinkle
    • 📄GDPR
    • 📄Privacy Policy
  • Release Notes
    • 📢Release Notes
      • 🗒️Release Notes - v12.1 (New)
      • 🗒️Release Notes - v12.0
      • 🗒️Release Notes - v11.0
      • 🗒️Release Notes - v10.8
      • 🗒️Release Notes - v10.7
      • 🗒️Release Notes - v10.6
      • 🗒️Release Notes - v10.5
      • 🗒️Release Notes - v10.4
      • 🗒️Release Notes - v10.3
      • 🗒️Release Notes - v10.2
      • 🗒️Release Notes - v10.1
      • 🗒️Release Notes - v10.0
      • 🗒️Release Notes - v9.31
      • 🗒️Release Notes - v9.30
      • 🗒️Release Notes - v9.29
      • 🗒️Release Notes - v9.28
      • 🗒️Release Notes - v9.27
      • 🗒️Release Notes - v9.25
      • 🗒️Release Notes - v9.24
      • 🗒️Release Notes - v9.23
      • 🗒️Release Notes - v9.22
      • 🗒️Release Notes - v9.21
      • 🗒️Release Notes - v9.20
      • 🗒️Release Notes - v9.19
      • 🗒️Release Notes - v9.18
      • 🗒️Release Notes - v9.17
      • 🗒️Release Notes - v9.16
      • 🗒️Release Notes - v9.14
      • 🗒️Release Notes - v9.13
      • 🗒️Release Notes - v9.12
      • 🗒️Release Notes -v9.8
      • 🗒️Release Notes - v9.7
      • 🗒️Release Notes - v9.6
      • 🗒️Release Notes - v9.5
      • 🗒️Release Notes - v9.4
      • 🗒️Release Notes - v9.3
      • 🗒️Release Notes - v9.2
      • 🗒️Release Notes - v9.1
      • 🗒️Release Notes - v9.0 (Major)
      • 🗒️Release Notes - v7.23
      • 🗒️Release Notes - v7.21
      • 🗒️Release Notes - v7.20
      • 🗒️Release Notes - v7.15
      • 🗒️Release Notes - v7.14
      • 🗒️Release Notes - v7.13
Powered by GitBook
On this page
  • Data Import Concepts
  • Step-by-Step Guide
  • STEP-1: Allow Mongo to accept connection from Sprinkle
  • STEP-2: Configure Mongo Connection
  • STEP-3: Configure Mongo data import
  • STEP-4: Create a Dataset
  • STEP-5: Run and schedule Ingestion
  1. Ingesting your data
  2. Data Imports
  3. Databases

Mongo DB

Guide to Integrate your Mongo DB with Sprinkle

PreviousGoogle BigQueryNextMySQL DB

Last updated 1 year ago

Data Import Concepts

Before setting up the data import, learn about data import concepts

Step-by-Step Guide

STEP-1: Allow Mongo to accept connection from Sprinkle

  1. Network Connectivity:

    • If Mongo server is on public network, accessible over public IP, allow inbound connection on mongo port (default is 27017) from Sprinkle IPs (34.93.254.126, 34.93.106.136)

    • If Mongo server is on private network, configure in Advanced Settings.

  2. Create a Read-Only user, providing any name like "sprinkle"

STEP-2: Configure Mongo Connection

To learn about Connection, refer

  • Log into the Sprinkle application

  • Navigate to Ingest -> Connections Tab -> Setup Connections ->

  • Select Mongo database

  • Provide all the mandatory details

    • Distinct Name: Name to identify this connection

    • Host: Provide the IP address or Host name.

    • Port: Provide the Port number.

    • Database: Provide a database name if there is any, it should be an existing database.

    • Username

    • Password

    • Advanced Settings: If Yes:-

      • Connection Scheme: Enter the connection Schema.

      • Connection Properties: You can provide optional connection properties. ex- key1=value1&key2=value2. The most common properties used for mongo is authSource=admin

      • Binlog Replication: If enabled, mongo binlog is used to fetch changed/new rows. This setting cannot be changed later once selected. For more details

      • Connect via SSH Host: If Yes:-

        • SSH Host: IP address or hostname of the SSH server.

        • SSH Public Key: Add this public key to the ~/.ssh/authorized_keys file on the ssh host machine.

        • SSH Login Username

  • Test Connection

  • Create

STEP-3: Configure Mongo data import

  • Navigate to Ingest -> Data Imports Tab -> Setup Sources ->

  • Select Mongo database

  • Provide the name -> Create

  • Connection Tab:

    • From the drop-down, select the name of the connection created in STEP-2

    • Update

STEP-4: Create a Dataset

  • Database Name (Required): Database name to select the collection from

  • Collection Name (Required): Collection to ingest

  • Destination Schema (Required): Data warehouse schema where the table will be ingested into

  • Destination Table name (Required): It is the table name to be created on the warehouse. If not given, sprinkle will create like ds_<datasourcename>_<tablename>.

  • Ingestion Mode: (Required)

    • Complete: Ingest full data from the source table in every ingestion job run. Choose this option if your table size is small (<1 million rows) and you want to ingest it infrequently (a few times a day)

    • Incremental: Ingest only the changed or inserted rows in every ingestion job run. Choose this option if your table size is large and you want to ingest in real-time mode.

    Time Column needs to be provided here

    Time Column Type:

  • Periodic Full Fetch (Required): If yes, periodically full table will be fetched from the source. Would be useful if data gets deleted from the source and you want to keep the data in sync with the source. If not, only incremental data will be pulled in every run.

    • Full Fetch Interval (Required): This will fetch complete rows in the table based on the interval selected. In this, an interval is according to the days of the week (including every night). For example, Every Sunday / Every Monday / Every Night, etc.

  • Automatic Schema (Required):

    • Yes: Schema is automatically discovered by Sprinkle (Recommended)

      • Flatten Level (Required): Select from One Level or Multi Level. In one level, flattening will not be applied on the complex type. They will be stored as a string. In multi-level, flattening will be applied in complex level till they become simple type.

    • No: Warehouse Schema to be provided Format for Warehouse schema is : Col1 datatype, Col2 datatype,Col3 datatype Datatype should be warehouse specific.

  • Filter: For Date column filter select the Date type and give the value in the string(format:yyyy-mm-dd) or long format. For other data type filter must give value in a proper format.

    • Date Type Filter, Other Date Type Filter:

    • No Filter: No filter is required

STEP-5: Run and schedule Ingestion

In the Ingestion Jobs tab:

  • Trigger the Job, using the Run button

  • To schedule, enable Auto-Run. Change the frequency if needed

To learn about data import, refer

Datasets Tab: To learn about Dataset, refer . Add a Dataset for each collection that you want to replicate, providing the following details:

Destination Create Table Clause: Provide additional clauses to warehouse-create table queries such as clustering, partitioning, and more, useful for optimizing DML statements. on how to use this field.

Time Column Name: Note: Time Column Type can be of three different data types:- String, Date, or Numeric. To Know more about

Filter JSON: Provide filter json enclosed in {}. like {"key1":{"$gte":value1},"key2":{"$gt":value2}}.

☄️
here
SSH Tunnel
here
see this
here
here
Learn more
Ingestion Modes, refer here
for more details