When copying data from Salesforce, you can use either SOQL query or SQL query. Note that these two has different syntax and functionality support, do not mix it. You are suggested to use the SOQL query, which is natively supported by Salesforce. Process for incremental loading, or delta loading, of data through a watermark:
Full Answer
How do I import data from Salesforce to hive?
To start the flow, starting with the QueryDatabasetable, right click every processor and start them. Once the processors are started you can see the data flow from Salesforce to Hive. Once the PutSQL processor has completed the process, you should now be able to find the data in your Hive instance.
How to copy data from Salesforce to another database?
When copying data from Salesforce, you can use either SOQL query or SQL query. Note that these two has different syntax and functionality support, do not mix it. You are suggested to use the SOQL query, which is natively supported by Salesforce.
How to create a NiFi flow in Salesforce using JDBC?
Set up the Salesforce JDBC driver for the Nifi Flow Run the Nifi if you haven’t started it yet. You can do this by running bin/nifi.sh Open your browser and go to http://localhost:8080/nifi. to access the Nifi UI, where you can create the flow. To start with building the flow, let’s start with configuring the JDBC drivers first.
How to connect to Salesforce using controller service?
Fill in the details to connect to Salesforce as below. Go to Settings tab and rename the controller service if you want to. I changed mine to SalesforceConnect and click on Apply button to save the settings. Now it’s time to enable this controller service.
Why You Should Avoid Full Reloads If Possible
Doing full reloads in cases where you could be doing incremental processing can potentially cause many problems in your organization.
Finding Integration Operations That Could Benefit From Incremental Processing
There are many scenarios where you might have created an operation that does full reloads of record sets. You should review the following examples to see if they match your operations, and if so, consider updating your operations to use incremental processing.
When Should You Use Incremental Processing?
You should use incremental processing whenever the Salesforce processing time to complete the integration data change is less than the processing time it would take to rewrite or reload the entire data set.
When Should You Use Full Rewrites?
There are scenarios where you won’t be able to use incremental processing. If the data changes affect all, or the majority of records for a particular object, you’ll probably need to do full reloads.
What are Some Ways to Apply Incremental Processing?
For inbound integration operations that are pushing data changes from an external system to Salesforce, here are some ways to use incremental processing.
Summary
On average, it is more common that data integration operations process a relatively small percentage of the overall data set involved. Because of this, if you have any integration operations that are doing full reloads, you should review these operations to see if you can utilize incremental processing.
Introduction
Apache Nifi is an open source project that was built for data flow automation and management between different systems. Some of the key features of Nifi, in addition to data flow, are ease of use with a drag and drop UI, easily scalable to run a single server or in a clustered mode across many servers.
Prerequisites
This tutorial assumes that you have Apache Nifi and a DataDirect JDBC driver. If you do not have those items, please follow these quick steps:
Add Drivers to the Nifi Classpath
Go to the install location of Progress DataDirect drivers, which is usually at
Set up the Hive JDBC driver for the Nifi Flow
Create another controller service for configuring the Hive connection. Repeat steps 4 through 9 above, except in step 7 substitute the details for the Hive driver, as follows:
Build the Nifi Flow
To build the Nifi Flow, first we would have to read the data from Salesforce and it can be done using a processor called QueryDatabaseTable which supports incremental pulling of the data.
Configure the ConvertJSONtoSQL Processor
Before going ahead, if you don’t have an opportunity table with the same schema as Salesforce, you would have to create it first. I have tried to find if there was an automated way of achieving this, but I couldn’t find any. If you find a way to do this let me know in comments.
Run the Flow
To start the flow, starting with the QueryDatabasetable, right click every processor and start them. Once the processors are started you can see the data flow from Salesforce to Hive. Once the PutSQL processor has completed the process, you should now be able to find the data in your Hive instance.