Now when we talk about Dataflows, the first thing that comes in our mind is the data flowing between differnt nodes, connecting different nodes and getting processed in between nodes (if required) and all other things happening.
Not to get Confused GPars Dataflow comes to the rescue. Now lets move on to DataFlows
Lets start with a simple demo:
Whenever the scenario is such that a task is dependent on another task then we can use Dataflows, and the tasks gets executed in another thread just like Actors
Here Task 2 (t2) depends in Task 1 (t1), which depends on Task 3 (t3) and Task 4(t4).
t2 waits for t1 and t1 eventually for t3 and t4 to complete the whole cycle.
Lets go through another example:
The two important inseparable aspects of Dataflow tasks are:
1) Operators
2) Selectors
Here is a simple example of a Dataflow Selector.
The concept behing Dataflow Selector is as follows:
A Selector starts processing as soon as it finds value in any of the available channels passed to it.
Let's probe it with the help of an example:
Interesting isn't it!!!!!
LIKE Selector, Dataflow Operator just acts in reverse to Selector principle.
Here, the operator will not execute until all the values are available in the specified channel(s), and once they are available, the associated closure gets invoked and we can also bind values explicitly to the output channel.
Let's see an example:
The operator waits for all the 3 channels to get the data and Task2 waits for the output channel to be populated.
So the Task1 executes, and it populates all the 3 channels.
Once all the 3 channels are populated, the operator which is in waiting status till now executes the closure and binds the required value to the output channel, which makes Task2 execute which has been waiting for the output channel to be populated.
So we can say that the Operator Synchronizes until values of all DataflowVariable becomes available and a Selector when any one of the channels has a value.
Lets move on to another example:
The Callaback methods within Listeners which gets executed at the onset/end of Opeartor's various LifeCycle Phases and are itself self explanatory.
Unlike previous example, we have used DataflowBroadcast and DataflowQueue instead of DataflowVariable.
There is a task populating DataflowQueue a,b & c with some delay. As soon as they are populated an operator acts on them and binds them manually (output channels represented by index ordering) to different output channels based on some condition.
There are other Selectors and Operators on those output channels which acts accordingly. The main objective behind this example is to demonstrate the usage of Listeners.
Not to get Confused GPars Dataflow comes to the rescue. Now lets move on to DataFlows
Lets start with a simple demo:
import groovyx.gpars.dataflow.Dataflows
import static groovyx.gpars.dataflow.Dataflow.task
class DataflowTest {
static main(args)
{
Dataflows data = new Dataflows()
def z
def t1 = task{
println "Sum Task is waiting (Thread: ${Thread.currentThread()})"
data.z = data.x+data.y
println "The Summation is $data.z (Thread: ${Thread.currentThread()})"
}
def t2 = task
{
println "Mul Task is waiting (Thread: ${Thread.currentThread()})"
def a = data.z * data.x
println "The Product is: $a"
}
def t3 = task{
println "Assigning x.... (Thread: ${Thread.currentThread()})"
data.x = 3
}
def t4 = task{
println "Assigning y.... (Thread: ${Thread.currentThread()})"
data.y = 7
}
println "The Main Thread is (Thread: ${Thread.currentThread()})"
[t2.join()]
}
}
Here we have defined tasks with the the help of Dataflow and some variables associated with Dataflow.Whenever the scenario is such that a task is dependent on another task then we can use Dataflows, and the tasks gets executed in another thread just like Actors
Here Task 2 (t2) depends in Task 1 (t1), which depends on Task 3 (t3) and Task 4(t4).
t2 waits for t1 and t1 eventually for t3 and t4 to complete the whole cycle.
Lets go through another example:
import groovyx.gpars.dataflow.Dataflows
import static groovyx.gpars.dataflow.Dataflow.task
class DataFlowTest2 {
static Dataflows data = new Dataflows()
static Thread t1,t2
static def t1()
{
def t1 = task{
println "Sum Task is waiting (Thread: ${Thread.currentThread()})"
def z = data.x+data.y
println "The Summation is...... $z (Thread: ${Thread.currentThread()})"
}
}
static def t2()
{
task{
println "Assignment by (Thread: ${Thread.currentThread()})"
try{
data.x = 100
data.y = 30
data.x = 1000
}
catch(e)
{
e.printStackTrace()
}
println "Assignment Done...."
}
}
static main(args)
{
t1()
t2()
Thread.currentThread().join()
}
}
This is also quite same as the previous one, the only difference is that the Dataflow Tasks we have defined within static methods. From the main we have invoked those static methods to initiate those tasks.The two important inseparable aspects of Dataflow tasks are:
1) Operators
2) Selectors
Here is a simple example of a Dataflow Selector.
The concept behing Dataflow Selector is as follows:
A Selector starts processing as soon as it finds value in any of the available channels passed to it.
Let's probe it with the help of an example:
import groovyx.gpars.dataflow.*
import static groovyx.gpars.dataflow.Dataflow.*
class DataFlowSelectTest {
static def a = new DataflowVariable()
static def b = new DataflowVariable()
static def c = new DataflowVariable()
static mains(args)
{
task{
println "Sleeping Task1"
sleep 3000
println "EndSleep Task1"
a << 2
}
task{
println "Sleeping Task2"
sleep 6000
println "EndSleep Task2"
b << 22
}
task{
println "Sleeping Task3"
sleep 5000
println "EndSleep Task3"
c << 222
}
/**
* Select waits for one of the Tasks to finish as there is wait
*/
def result = select([a, b, c])
println "The fastest result is class ${result.getClass()} ------- ${result([false,true,true]).value}"
def oper = '+'
def num1 = 11
def num2 = 22
def formula= "$num1 $oper $num2"
println "$formula: ${new GroovyShell().evaluate(formula)}"
}
}
So in the above example we can see that we have defined 3 tasks each populating a DataflowVariable with some delay and there is a Selector, which is acting on these 3 DataflowVariable(s), and as soon as any one of the DataflowVariable(s) becomes populated the Selector executes and it populates the result with the value of the DataflowVarible being populated first.Interesting isn't it!!!!!
LIKE Selector, Dataflow Operator just acts in reverse to Selector principle.
Here, the operator will not execute until all the values are available in the specified channel(s), and once they are available, the associated closure gets invoked and we can also bind values explicitly to the output channel.
Let's see an example:
import static groovyx.gpars.dataflow.Dataflow.*
import groovyx.gpars.dataflow.*
public class DataFlowOperator {
static DataflowQueue a = new DataflowQueue()
static DataflowQueue b = new DataflowQueue()
static DataflowQueue c = new DataflowQueue()
static DataflowQueue d = new DataflowQueue()
static mains(args)
{
println "By Thread ${Thread.currentThread()}"
try{
operator(inputs: [a, b, c], outputs: [d]) {x, y, z ->
println "x:$x \n y:$y \n z:$z By Thread ${Thread.currentThread()}"
bindAllOutputValues x+y+z
}
task{
println "In Assignment Task.....By Thread ${Thread.currentThread()}"
a << 10
b << 20
c << 30
println "End Of Assignment Task.....By Thread ${Thread.currentThread()}"
}
def t = task{
println "The value of d is By Thread ${Thread.currentThread()}"
def x=d.val
println "The value of x is $x By Thread ${Thread.currentThread()}"
}
Thread.sleep(10000)
}
catch(e)
{
e.printStackTrace()
}
println "End of Main........ By Thread ${Thread.currentThread()}"
}
}
Here we have defined two tasks and an operator.The operator waits for all the 3 channels to get the data and Task2 waits for the output channel to be populated.
So the Task1 executes, and it populates all the 3 channels.
Once all the 3 channels are populated, the operator which is in waiting status till now executes the closure and binds the required value to the output channel, which makes Task2 execute which has been waiting for the output channel to be populated.
So we can say that the Operator Synchronizes until values of all DataflowVariable becomes available and a Selector when any one of the channels has a value.
Lets move on to another example:
import java.util.concurrent.TimeUnit;
import groovyx.gpars.dataflow.DataflowQueue;
import static groovyx.gpars.dataflow.Dataflow.*
import groovyx.gpars.dataflow.*
import groovyx.gpars.dataflow.operator.DataflowEventListener;
import groovyx.gpars.dataflow.operator.DataflowProcessor;
class DataFlowOperatorTest {
static DataflowQueue a = new DataflowQueue()
static DataflowQueue b = new DataflowQueue()
static DataflowQueue c = new DataflowQueue()
/**
* When we will be using the Broadcast then
* we need to comment the Queue Block and open up
* the Broadcast block.
*
* Cause we have used the same DataFlow entities in both
* Operators and Selectors so if we use Queues, the Operators
* will be in effect, the Selectors will not be in action, as in
* case of Queues it can be used with either Operator or Selector.
* In order to use for both Operators and Selectors Broadcasts
* should be used, so that they can be registered with multiple
* operational entities.....
*
*/
static DataflowBroadcast d = new DataflowBroadcast()
static DataflowBroadcast e = new DataflowBroadcast()
/*static DataflowQueue d = new DataflowQueue()
static DataflowQueue e = new DataflowQueue()*/
static DataflowQueue highVolumeQueue = new DataflowQueue()
static def listnrs = new DataflowEventsListenerImpl()
static def ChanlListener = new DataflowChannelListenerImpl()
static mains(args)
{
println "By Thread ${Thread.currentThread()}"
highVolumeQueue.getEventManager().addDataflowChannelListener(ChanlListener)
try{
/**
* Block for Broadcast
*
*/
def op1 = operator(inputs:[d.createReadChannel(), e.createReadChannel()]) { val1, val2 ->
/**
* The delegate inside Operators/Selectors is
* DataflowProcessor
*
*/
registerChannelListenersToAllInputs(ChanlListener)
println "***** The value is of d N e is: $val1 N $val2"
}
def selector1 = selector(inputs : [d.createReadChannel(), e.createReadChannel()]) {val, idx ->
println "%%%%%%% The SELECTOR Val is: $val \n Index: $idx \n"
}
/**
* Within Selectors we can specify outputs too just like normal Operators.
* For the sake of simplicity we have omitted that.
*
*/
/**
* End of Block for Broadcast
*/
/**
* Block for Queue
*
*/
/*def op1 = operator(inputs:[d, e]) { val1, val2 ->
println "***** The value is of d N e is: $val1 N $val2"
}
def selector1 = selector(inputs : [d, e]) {dVal, eVal ->
println "%%%%%%% The SELECTOR dVal is: $dVal \n eVal: $eVal \n"
}*/
/**
* End of Block for Queue
*/
/*def op2 = operator(inputs:[e]) { val ->
println "The value is of e: $val"
}*/
def op3 = operator(inputs:[highVolumeQueue]) { highVal ->
println "The High vOlume DATA: $highVal"
}
operator(inputs: [a, b, c], outputs: [d,e,highVolumeQueue], listeners: [listnrs]) {x, y, z ->
println "x:$x \n y:$y \n z:$z \n By Thread ${Thread.currentThread().isDaemon()}"
println "===== Binding First Output ====="
bindOutput 1, x+y+z
TimeUnit.MILLISECONDS.sleep(5000)
println "===== Binding Second Output ====="
bindOutput 0, x+z
if(x+2*y+z >= 80 )
bindOutput(2,fireCustomEvent(x+2*y+z))
else
bindAllOutputs x+2*y+z
}
task{
println "In Assignment Task.....By Thread ${Thread.currentThread()}"
a << 10
b << 20
TimeUnit.MILLISECONDS.sleep(6000)
c << 30
println "End Of Assignment Task.....By Thread ${Thread.currentThread()}"
}
task
{
println "Doing some Stupidity in the mean time..... :-) :-)"
}
[op1,op3]*.join()
/**
* Termination of the Operators/Selectors can be done with
* terminate(), terminateAfterNextRun()
* These methods call the Life Cycle Event Methods of the registered Life Cycle Event Handlers of the
* corresponding Selectors/Operators.
*
* Apart from these we can also use PoisonPill Control Message,
* which needs to be passed to the inputs of the Operators/Selectors
* i.e. Channels/Broadcasts/Variable etc and Event Methods of the registered Life Cycle Event Handlers
* gets invoked.
*
*/
}
catch(e)
{
e.printStackTrace()
}
println "End of Main........ By Thread ${Thread.currentThread()}"
}
}
class DataflowEventsListenerImpl implements DataflowEventListener {
public void registered(DataflowProcessor processor) {
// TODO Auto-generated method stub
println "******* Registered ...."
}
public void afterStart(DataflowProcessor processor) {
// TODO Auto-generated method stub
println "******* afterStart ...."
}
public void afterStop(DataflowProcessor processor) {
// TODO Auto-generated method stub
println "******* afterStop ...."
}
public boolean onException(DataflowProcessor processor, Throwable e) {
// TODO Auto-generated method stub
println "******* onException1 .... ${e.printStackTrace()}"
return false;
}
public Object messageArrived(DataflowProcessor processor,
DataflowReadChannel<Object> channel, int index, Object message) {
// TODO Auto-generated method stub
println "******* messageArrived .... $message for Channel $channel at Index $index"
return message;
}
public Object controlMessageArrived(DataflowProcessor processor,
DataflowReadChannel<Object> channel, int index, Object message) {
// TODO Auto-generated method stub
println "******* controlMessageArrived .... $message for Channel $channel at Index $index"
return message;
}
public Object messageSentOut(DataflowProcessor processor,
DataflowWriteChannel<Object> channel, int index, Object message) {
// TODO Auto-generated method stub
println "******* messageSentOut .... $message for Channel $channel at Index $index"
return message;
}
public List<Object> beforeRun(DataflowProcessor processor,
List<Object> messages) {
println "******* beforeRun .... messages $messages"
// TODO Auto-generated method stub
return messages;
}
public void afterRun(DataflowProcessor processor, List<Object> messages) {
// TODO Auto-generated method stub
println "******* afterRun .... messages $messages"
}
public Object customEvent(DataflowProcessor processor, Object data) {
// TODO Auto-generated method stub
println "******* customEvent .... for Data: $data"
return 'I am Firing Custom Event';
}
}
class DataflowChannelListenerImpl implements DataflowChannelListener
{
public void onMessage(Object message) {
// TODO Auto-generated method stub
println "@@@@@@@ Channel Listener invoked with Input"
}
}
In this code snippet we have introduced Listeners for Operators which responds to Opeartor's LifeCycle Phases and Events and another Listener for DataflowQueue.The Callaback methods within Listeners which gets executed at the onset/end of Opeartor's various LifeCycle Phases and are itself self explanatory.
Unlike previous example, we have used DataflowBroadcast and DataflowQueue instead of DataflowVariable.
There is a task populating DataflowQueue a,b & c with some delay. As soon as they are populated an operator acts on them and binds them manually (output channels represented by index ordering) to different output channels based on some condition.
There are other Selectors and Operators on those output channels which acts accordingly. The main objective behind this example is to demonstrate the usage of Listeners.


No comments:
Post a Comment