Data ingestion can be done in two ways.
- Complete ingestion mode.
- Incremental ingestion mode.
Complete ingestion mode
Complete ingestion loads the entire data at once irrespective of the pre-existing data. This takes significant time and also increases load on the database server.
While adding table to data source you can select type of ingestion mode.
Incremental ingestion mode
In Incremental loading only new and latest data is ingested to the data source. In this process three tables are created.
ds_datasource_table_delta. Every night
table_base goes under complete loading and during the day
table_delta pulls data based on updated_atcolumn from midnight (last checkpoint of complete loading of
table_base) to till now. Finally creating
ds_datasource_table by merging these two tables and removing duplicates using one or more unique keys. However, past data is already present in
table_base so it tries to append only latest data from
table_delta. Table should have relevant time column (
updated_at) for incremental loading.
Select updated Rows, Time column and one or more unique keys.