Skip to main content

RPA integration with UI.Vision - Temporal.io based Worker

This tutorial in a nutshell​

In this example, we will build a Temporal-based worker that will allow us to run a robot built using UI.Vision.

What is UI.Vision?​

It is an RPA framework that allows us to automate operations in web and desktop applications. It is also multiplatform, being able to run on Windows, Linux and Mac. One of its strengths is that it allows us to perform image and text recognition operations through automated visual tests using a user interface.

But in this example, what interests us is how to integrate it with KuFlow and take advantage of the benefits offered by our platform such as the interaction of this robot with task flows performed by humans. In order to keep the example as simple as possible, we will leave to the user the choice of creating a more complicated workflow with interaction of different tasks and users, and we will focus on running a workflow without human interaction and with a single task whose responsible will be our robot. If you wish, in other examples and documentation on this website, you can see how to orchestrate more complicated flows.

In essence, the robot will take a screenshot of our desktop and upload it to a KuFlow task.

We will choose Java for the implementation of the worker, but remember that this implementation can be done in other programming languages.

Prerequisites​

As always, you just need to create a FREE user account in KuFlow (app.kuflow.com) if you don't already have one. Likewise, if you are not yet familiar with our APP, our youtube channel has many videos that will help you in the task.

Create the credentials for the Worker​

We will configure an APPLICATION that will provide us with the necessary credentials so that different external systems can interface with KuFlow, in this example, the necessary credentials for our Worker.

Go to the Settings > Applications menu and click on Add application. We establish the name we want and save. Next, you will get the first data needed to configure our Worker.

  • Identifier: Unique identifier of the application.
    • Later in this tutorial, we will configure it in the kuflow.api.client-id property of our example.
  • Token: Password for the application.

    • Later in this tutorial, we will configure it in the kuflow.api.client-secret property of our example.

Create the process definition​

We need to create the definition of the process that will execute our Workflow. In this section we will configure the KuFlow tasks of which it is made up as well as the information necessary to complete said tasks, the process access rules (i.e. RBAC), as well as another series of information. To do this we go to the Setting > Processes menu and create a new process. You can take a quick look at this video to get an idea of the user interface for process definition.

A Process Definition with the following data:

  • Description
    • Free text description about the Workflow.
  • Workflow
    • Workflow Engine
      • KuFlow Engine, because we are designing a Temporal-based Worker.
    • Workflow Application
      • The application to which our Worker will connect to
    • Task queue
      • The name of the Temporal queue where the KuFlow tasks will be set. You can choose any name, later you will set this same name in the appropriate configuration in your Worker. For example: UIVisionQueue.
    • Name
      • It must match the name of the Java interface of the Workflow. In our example, UIVisionSampleWorkflow is the name you should type in this input.
  • Permissions
    1. At least one user or group of users with the role of INITIATOR in order to instantiate the process through the application.
      1. Optional: In order to view administrative actions in the GUI, it may interesting to set the Manager role as well for some user.

A Task Definition in the process with the following data:

  • Task "My robot results"
    • Description: Free text description about the Task.
    • Code: ROBOT_RESULTS
    • Candidates:
      • This is an automated task, i.e. a system (in this case, our robot) will be in charge of performing it. Therefore, the application we have configured is indicated here as a candidate.
    • Element to fill the email:
      • Name: Field's label, for example "Screenshot"
      • Code: SCREENSHOT
      • Type: Field
      • Properties: Mandatory, checked.
      • Field Type: Document

Source code for our Worker​

The source code for this example can be obtained from our java examples repository

However, an alternative methodology could be to download the example template code for the process we are going to implement and compare it with the complete example in our repository. To download the initial template you must access the workflow definition in the App, and once configured, use the download template option.

Main technologies used in the example​

This code will serve as a starting point for implementing our Loan Application worker. The requirements for its use are the following:

  • Java JDK
    • You need to have a Java JDK installed on your system. The current example code uses version 17, but is not required for the KuFlow SDK. You can use for example Adoptium distribution or any other. We recommend you to use a tool like SDKMan to install Java SDK distributions in a comfortable way.
  • Maven
    • To build the example. It is distributed in an integrated way so it is not necessary to install it in isolation.
  • Spring Boot and Spring Cloud
    • To wire all our integrations.
    • Note: There is a version of this example that does not use spring available in our repository.
  • IDE
    • An IDE with good Java support is necessary to work comfortably. You can use VSCode, IntelliJ Idea, Eclipse or any other with corresponding Java plugins.

To make things simpler, the following dependencies have been used in our example:

  • KuFlow Java SDK

    • Provide some activities and utilities to work with KuFlow.

  • Temporal Java SDK

    • To perform GRPC communications with the KuFlow temporal service.

  • KuFlow CLI (kuflowctl)

    • A command line tool that allows you to interact with the KuFlow Rest API.

  • UI.Vision

    • RPA framework that allows us to automate operations in web and desktop applications

Implementation details​

The workflow​

The entry point to the Workflow execution is determined by the @WorkflowMethod annotation, in our code the main method would be the following:

UIVisionSampleWorkflowImpl.java

@Override
public WorkflowResponse runWorkflow(WorkflowRequest workflowRequest) {
    this.kuflowGenerator = new KuFlowGenerator(workflowRequest.getProcessId());

    this.createTaskRobotResults(workflowRequest);

    LOGGER.info("UiVision process finished. {}", workflowRequest.getProcessId());

    return this.completeWorkflow(workflowRequest);
}

The structure is very simple:

  1. Initialize a identifier generator in order to use deterministic Id in idempotent calls.
  2. Create a KuFlow Task to execute the bot.
  3. Report completed Workflow.

Here the most relevant thing is the method that the robot executes. The orchestration is simple:

  1. Create task
  2. Claim the task
  3. Execute the robot
  4. Complete the task
private void createTaskRobotResults(WorkflowRequest workflowRequest) {
    UUID taskId = this.kuflowGenerator.randomUUID();

    // Create task in KuFlow
    TaskDefinitionSummary tasksDefinition = new TaskDefinitionSummary();
    tasksDefinition.setCode(TASK_ROBOT_RESULTS);

    Task task = new Task();
    task.setId(taskId);
    task.setProcessId(workflowRequest.getProcessId());
    task.setTaskDefinition(tasksDefinition);

    CreateTaskRequest createTaskRequest = new CreateTaskRequest();
    createTaskRequest.setTask(task);
    this.kuFlowActivities.createTask(createTaskRequest);

    // Claim task by the worker because is a valid candidate.
    // We could also claim it by specifying the "owner" in the above creation call.
    // We use the same application for the worker and for the robot.
    ClaimTaskRequest claimTaskRequest = new ClaimTaskRequest();
    claimTaskRequest.setTaskId(taskId);
    this.kuFlowActivities.claimTask(claimTaskRequest);

    // Executes the Temporal activity to run the robot.
    ExecuteUIVisionMacroRequest executeUIVisionMacroRequest = new ExecuteUIVisionMacroRequest();
    executeUIVisionMacroRequest.setTaskId(taskId);
    this.uiVisionActivities.executeUIVisionMacro(executeUIVisionMacroRequest);

    // Complete the task.
    CompleteTaskRequest completeTaskRequest = new CompleteTaskRequest();
    completeTaskRequest.setTaskId(taskId);
    this.kuFlowActivities.completeTask(completeTaskRequest);
}

But, wait a minute, where is the implementation of the Temporal activity that the UI.Vision robot executes?

The activities​

Well, in the KuFlow SDK we have an implementation of this activity ready to use. Its operation is simple. Basically what you do is to run an external process installed on the machine, which in this case is none other than a Chrome browser with the UI.Vision plugin configured and a robot registered to it. If this implementation does not fit your needs, you can always make your own using the provided activity as a reference.

To learn more about this implementation, see the section on UI.Vision.

The robot​

At this point, we assume that you have the knowledge to implement a robot in UI.Vision, otherwise the documentation available on their website and their forum are of great help. Our robot will simply take a screenshot of our desktop, and upload it to the KuFlow task we have created in our workflow. The robot specification is the following:

CommandTargetValue
1captureDesktopScreenshot/tmp/kuflow_image.png
2XRunAndWaitkuflowctl--silent save-element-document --task-id=${!cmd_var1} --element-code=SCREENSHOT /tmp/kuflow_image.png
3store${!xrun_exitcode}result
4echo"Exit code: ${result}"
5assertresult0

Interaction with external processes from a UI.Vision robot is limited. However, it allows us to run external processes and get their output code. This is enough for our robot to interact with the KuFlow API and upload the results in the task. To do this we will use our CLI (kufloctl) which implements a client of our API, saving us from having to create our own implementation to upload the capture to KuFlow. This example assumes that you have this CLI installed on your system and configured to access with the credentials of the same application used in this tutorial. More details on how to do this can be found in the CLI documentation. Depending on how you decide to authenticate the CLI, you may need to adjust the value of command #2 to pass them the appropriate credentials.

IMPORTANT

The supplied macro code assumes that the CLI authenticates with a configuration file located in the default path.

This example assumes that you have installed UI.Vision with its Xmodules module, in the Google Chrome browser. Please make sure of it.

IMPORTANT

Make sure that in the extension details, the "Allow access to file URLs" option is checked.

Import the macro of our robot in the UI.Vision gui (Import JSON action). For this example, make sure it is done in the Storage Mode: File system (on hard drive), in order to store the results of the robot (the capture and logs) on the hard drive and not in the browser storage.

Some important details about the robot instructions:

  • Our activity passes to the robot the taskId of the task in which it should upload the capture taken.
  • To identify if the robot execution has finished successfully, we need to check the CLI response code (command #5), otherwise the robot execution would always finish successfully even if the command execution was unsuccessful.

The worker​

We will use two facades of activities from our component catalog to implement the activities that perform the communication with the KuFlow REST API and the UIVision activities. These activities along with the implemented Workflow, we register them in the Temporal bootstrap class.

TemporalBootstrap.java

private void startWorkers() {
    this.kuFlowTemporalConnection.configureWorker(builder ->
            builder
                .withTaskQueue(this.sampleEngineWorkerUiVisionProperties.getTemporal().getKuflowQueue())
                .withWorkflowImplementationTypes(UIVisionSampleWorkflowImpl.class)
                .withActivitiesImplementations(this.kuFlowActivities)
                .withActivitiesImplementations(this.uiVisionActivities)
        );

    this.kuFlowTemporalConnection.start();
}

Worker configuration​

Before starting our Worker for the first time, it is necessary to configure a series of properties. For simplicity, an application-fillme.yml file has been created where you need to fill in all these settings.

You must complete all the settings and replace the example values. The appropriate values can be obtained from the KuFlow application. Check the Prerequisites section of this guide.

# ===================================================================
# PLEASE COMPLETE ALL CONFIGURATIONS BEFORE STARTING THE WORKER
# ===================================================================

kuflow:
  api:
    # ID of the APPLICATION configured in KUFLOW.
    # Get it in "Application details" in the Kuflow APP.
    client-id: FILL_ME

    # TOKEN of the APPLICATION configured in KUFLOW.
    # Get it in "Application details" in the Kuflow APP.
    client-secret: FILL_ME

  activity:
    ui-vision:
      # Browser with UI.VISION pluging.
      # Example: /user/bin/google-chrome
      launch-command: FILL_ME

      # A directory where the robot can set its logs.
      # Example: /home/user/logs
      log-directory: FILL_ME

      # Path to the UI.VISION autorun html
      # Example: /home/user/ui.vision.html
      # See in: kuflow-samples-temporal-uivision-spring/etc/autostarthtml/ui.vision.html
      auto-run-html: FILL_ME

      # UI.Vision macro to run
      # Example: KuFlowScreenshot.json
      # see in: kuflow-samples-temporal-uivision-spring/etc/macro/KuFlowScreenshot.json
      macro: FILL_ME

      # Close browser when the macro is completed.
      # Optional:
      # closeBrowser: true

      # Close UI.Vision RPA when the macro is completed.
      # Optional:
      # close-rpa: true

      # It should be less than the duration specified in the StartToCloseTimeout of the UI.Vision Temporal activity.
      execution-timeout: 20m

application:
  temporal:
    # Temporal Queue. Configure it in the "Process definition" in the KUFLOW APP.
    kuflow-queue: FILL_ME

Please note that this is a YAML, respect the indentation.

Details​

The file to be configured in the auto-run-html property can be obtained from the repo, or it can be generated directly from the UI.Vision interface, by right-clicking on the macro (KuFlowScreenshot) and selecting the Create autorun HTML action.

Run​

Start the Worker. Normally, you will do this through the IDE. But if you prefer the console line, in the repository root folder type:

./mvnw spring-boot:run -f kuflow-samples-temporal-uivision-spring/pom.xml

If the Worker starts successfully, you will see the following log output.

----------------------------------------------------------
        Application 'KuFlow Engine Worker Sample UiVision' is running!
        Profile(s):     [default]
----------------------------------------------------------

When a connected Worker that implements our example process is registered, in the application we can see how the badge associated with the Workflow Engine field (in the Kuflow GUI) changes from Disconnected to Connected. Finally, users with the appropriate permissions will be able to run a Process that executes the Workflow we have just implemented. To do this, from the non-administrative part of the KuFlow application, we instantiate the process by clicking on the "Start process" button.

Summary​

In this tutorial we have shown how to run an RPA process implemented in UI.Vision through a Temporal.IO activity whose code is available in the KuFlow open source repositories. We have also used the command line client provided by KuFlow for the interaction of the robot with the KuFlow Rest API.

Kuflow Logo
Legal
Copyright © 2024 KUFLOW, S.L.