Overview
The Boomi Web Services Server connector is a listener that accepts HTTP and SOAP requests in real-time and initiates Dell Boomi AtomSphere processes. When a process containing this connector is deployed to an Atom, the Atom’s internal web server will listen for documents based on configurations made in the Web Services Server operation.
This connector has multiple uses like web application updating/deleting/ inserting records in database, database event triggering notification to user, salesforce to database and so on. This blog explores implementing this connector for extracting, transforming & loading (ETL) data from transactional database to data warehouse at real time. It is assumed the user is familiar with boomi concepts, building basic processes, database connectors & map functions.
Configuration for ETL data from OLTP database to Data Warehouse
As shown in figure-1 SQL triggers are fired when there is any modification to the OLTP database. The SQL trigger calls the boomi web services server connector which then initiates the boomi process. The boomi process connects to the OLTP database gets the changes to data and applies these changes to the Data Warehouse. With multiple insert/delete/updates happening on the transaction database, multiple boomi process are simultaneously executing to ELT data in the data warehouse.
Figure 1. Test configuration for ETL data from OLTP database to Data Warehouse
Image may be NSFW.
Clik here to view.
SQL trigger
For our use case lets consider a table in the OLTP database which when modified would trigger changes to the fact & destination tables in the Data Warehouse. The code for the SQL trigger is as follows:
CREATE TRIGGER [dbo].[trigger_name]
ON [dbo].[table_name]
AFTER INSERT
AS
BEGIN
--Declare all the variables
declare @object int
declare @return int
declare @valid int
declare @uri varchar(2000)
--Set the variables
set @valid = 0
--Delete all records in the temporary table
DELETE FROM tempins
--Insert the primary key of the inserted/deleted/updated records in a temporary table
INSERT INTO [dbo].[tempins](CA_ID)
SELECT CA_ID from inserted
--Url of web services server connector. For more information on building the url check the section “Getting the URL that your SQL trigger calls”
set @uri = 'http://<ipaddress OLTP server:port>/ws/simple/<Object name>?type=Insert'
--create the XMLHTTP object
exec @return = sp_oacreate 'MSXML2.ServerXMLHTTP', @object output
if @return = 0
begin
--Open the connection
exec @return = sp_oamethod @object, 'open', NULL, 'POST', @uri, false
if @return = 0
begin
--Send the request
exec @return = sp_oamethod @object, 'Send' , NULL
PRINT @return
end
if @return = 0
begin
declare @output int
exec @return = sp_oamethod @object, 'status', @output output
if @output = 200
begin
set @valid = 1
end
end
else
PRINT 'no output'
end
else
print 'no xmlhttpobject'
--Destroy the object
exec sp_oadestroy @object
if @valid = 1
print 'valid'
else
print
'invalid'
END
GO
Boomi Process
Figure-2 shows the boomi process which starts once the SQL trigger executes. The Web services server connector initiates the process. The database connector connects to the OLTP database gets the ids from the temp tables and queries the related fact and destination tables to get all the details of the records. The map shape maps these fields to the fact and destination fields in the Data Warehouse and changes are applied. For more information on building Boomi processes and getting started refer to the video http://www.youtube.com/watch?v=ZAb9Ev3tuMw&feature=youtu.be or the boomi help link http://help.boomi.com/atomsphere/GUID-B522EE93-E8A2-43CC-9D3E-EF37371AEF32.html
Figure 2. Boomi ETL of data from OLTP database to Data Warehouse based on specific event
Image may be NSFW.
Clik here to view.
The sections below describe the steps to follow specifically when using the web services connector.
Getting the URL that your SQL trigger calls
1. To get the url to ping from the SQL trigger, navigate to Manage Atom Management in your Boomi account.
2. Select the atom which is created in the network where the database resides. In the Atom properties window, select Shared Web Server settings option as shown in figure-3.
Figure 3. Shared Web Server Settings
Image may be NSFW.
Clik here to view.
3. Navigate to the user management window as shown in figure- 4. Copy the url displayed in the Base URL field. It should be in the form : 'http://<ipaddress OLTP server or servername:port>
Figure 4. User Managemment
Image may be NSFW.
Clik here to view.
4. Navigate to the web services server operation and copy the simple url path as shown in figure-5 Add it to the url in step 3 to get the complete url that the SQL trigger calls 'http://<ipaddress OLTP server or servername:port>/ws/simple/<object name>
Figure 5. Web Services Server Operation
Image may be NSFW.
Clik here to view.
5. Optionally to pass parameters in the url use ‘?’ followed by the parameter name=value. For e.g. 'http://<ipaddress OLTP server:port>/ws/simple/<Object name>?param1=Insert¶m2='
For more information on the boomi web service connector watch the videos: http://training.boomi.com/display/TDASH/Training+Video+Library
Accessing the parameters in Boomi process
Any parameters passed in the url can be accessed in the boomi process as “Dynamic Process Property” with query_ added before the parameter name as shown in figure-6
Figure 6. Dynamic Process Property
Image may be NSFW.
Clik here to view.
Performance
For a simple scenario of inserting/updating/deleting data from a table in the transaction database and in turn insert/delete/update on the fact & dimension table in the data warehouse had the following average response time:
- 4-5 sec for insert/update/delete for 1 to 15 rows
- 80.458 sec ( 1.3 min) for inserting 55000 rows
- 350.857 sec ( 5.8 min) for updating 55000 rows
- 591.532 sec ( 9.8 min) for deleting 55000 rows
The insert/update/delete statements were executes immediately. Three boomi processes were called and were executing simultaneously. The delete possibly took more time as it was waiting for SQL server to complete the updates before deleteing the records.
Figure 7. Average Response Time for insert/update/delete for 1 to 15 rows
Image may be NSFW.
Clik here to view.
Figure 8. Average Response Time for insert/update/delete for 55000 rows
Image may be NSFW.
Clik here to view.
Conclusion
In todays complex service-oriented architecture, bringing data together and making sense of it becomes increasingly difficult. Real-time data integration is being increasingly used by companies to improve their business and better serve their customers.The challenge with traditional batch systems is that processes are executed usually once daily and invariably overnight to reduce load on the architecture. The advantages of transitioning to a near-real-time system from the traditional nightly batch processing include improved knowledge of how business is performing and offering services that outperform competitors.
The procedure described in this blog shows how Boomi can be utilized in a near real-time information and on-demand ETL processing that can have a huge payback for certain business scenarios. Depending on the complexity of the target database, creating and managing all the database triggers and corresponding database processes can increase the overhead to maintain operations and potentially could have an impact on the overall performance.
Image may be NSFW.Clik here to view.