Skip to main content

KuBot desktop screenshot

This tutorial in a nutshell​

This tutorial presents a good starting point for implementing your first KuBot.

We will model two simple robots whose task will be to take a desktop screenshot of the user who is running the assisted robot. In the first one we will only take the screenshot and write it to the "home" folder on the user's computer. The second robot will be an evolution of the first where we will change the destination of the capture so that it is a KuFlow task and therefore have the capture in the KuFlow Cloud app.

This tutorial does not focus on the implementation of a workflow for our business process, for this you can consult any of our tutorials related to process implementations. In this article we will build a robot that assists the human in carrying out a task within a process. Therefore, the robot can be used in any process with human tasks, taking advantage of the independence and polyglot robustness that KuFlow offers when implementing business processes.

We will use Python as a programming language and we will not integrate any specific RPA building framework. This basic example is intended to introduce the developer to the world of assisted RPA.

Prerequisites​

Before you start you need to create a user in the app if you don't already have one. We must register it in KuFlow (app.kuflow.com). You will also need to have the Organization Administrator role since we are going to carry out administrative actions (definition of a process, definition of a KuBot...). To check this, click on the "settings" menu and see if you have access to the process definition menu.. If you do not see this menu, you probably do not have the necessary permissions to define processes, so you will have to request it from your Organization administrators, or simply create a FREE new account.

It is also recommended that you first read the general details about Kubots development in our documentation.

Source code for our Worker​

The source code for this example can be obtained from our python examples repository.

Main technologies used in the example​

The requirements to complete this tutorial are as follows:

  • Poetry (>=1.4.0)
    • For python packaging and dependency management
  • Python (>=3.11)
    • To build the example. It is distributed in an integrated way so it is not necessary to install it in isolation.
  • IDE
    • An IDE with good Python support is necessary to work comfortably. You can use VSCode, IntelliJ Idea, Eclipse or any other with corresponding Python plugins.

To make things simpler, the following dependencies have been used in our example:

  • KuFlow Python SDK
    • Provide a rest client, activities and utilities to work with KuFlow.

Implementation details​

Process workflow​

In order to use the KuBot that we will develop, we need to create a process with at least one human task that will assist. We will model the simplest case since our purpose is to explore the development of the KuBot and not the process workflow. We will model a process, with a single task.

  1. The user will start the process.
  2. The workflow will create a task that will contain a form (structure data) that will present a document, our desktop screenshot.
  3. The person responsible for this task will be the user initiating the process (human task).
  4. In a normal human task, the user would have to manually upload the screenshot taken. In our final solution, the KuBot will perform this step automatically.

Process settings​

  • Name

    • Users will locate it in the app by this name.

  • Description

    • Free text description about the Workflow.

  • Workflow

    • Workflow Engine

      • KuFlow Diagram Tool, we will use the serverless engine for simplicity.

    • Workflow Diagram

      • Leave empty for now (first, define the rest of the settings, such as tasks, then come back here)

  • Permissions

    • At least, one user or group of users must have the role of INITIATOR in order to instantiate the process through the application. In this tutorial, we will allow all users from this organization to be able to process initiation (Groups > Users)
    • At least one user must be the ADMINISTRATOR of the process, so we give our KuFlow user the role. Incidentally, we also give you the PROCESS MANAGER role so that you can manage the instances of this process of other users.

  • Structure data

    • For now we will create a structure that we will leave empty, throughout the tutorial we will return and complete it.
    • Name: Image form
    • Code: SCREENSHOT_FORM (or any other, we will not use it in this tutorial)
    • Form definition: Leave empty for now

  • Tasks

    • Task "Take screenshot"
      • Description: A screenshot must be provided
      • Code: SCREENSHOT_TASK
      • Structure data: Set the previous structure data created
      • Assisted robot: Leave empty for now
      • Permissions:
        • Roles > Process Initiator as Candidate (it is a human task that will be performed by the process initiator himself, therefore we select the Process Initiator role)

At this point, edit the process and open the workflow editor and leave it as follows:

  • One "Start" node
  • One "Task" node with our task definition. Set ${.initiator.id} expression in Owner input.
  • One "Stop" node

Save the changes.

KuBot​

In this tutorial we will use VSCode in our development. For greater agility, we will clone the example repo and explain the important parts. Of course, you can structure the Python module however you like. We will use Poetry in this example but finally the KuBot package code that we will upload to our platform will only contain a manifest, the code and a "requirements.txt" file with the necessary dependencies. We will not upload our own Poetry files.

  1. Open a terminal and make sure you have pyhton and poetry available in it.
  2. Clone the repo: https://github.com/kuflow/kuflow-samples-python
  3. Move to repo directory and execute code .
  4. The cloned repository contains different examples, we will focus on the "kuflow-samples-kubot-desktop-screenshot" folder that contains the example of this tutorial. We open a Terminal integrated in VSCode and move to said directory. Make sure that you do not have any virtual env already activated in said terminal since we want to create a new one (test with poetry env info, If there is no virtual environment activated, the result will indicate that a virtual environment was not found).
    • cd kuflow-samples-kubot-desktop-screenshot
  5. Next we are going to build the virtual environment with the dependencies of our project. For this we will use Poetry. Examine the "pyproject.toml" file to get a first look at the module structure and its dependencies. The execute:
    • poetry install
  6. Instructs VSCode to use the created virtual env. To do this, we open the command palette and select the Python interpreter that is in the virtual environment that has been built in a hidden directory.
    1. Open command palette, search "Python: select interpreter"
    2. Locate and select the python executable in the environment. For example: kuflow-samples-kubot-desktop-screenshot/.venv/bin/python

Now we are ready to write our RPA code. We'll start by writing some very simple Python code that simply takes a screenshot and sets it to the user's Home folder. We have written this code for you. Open:

kuflow-samples-kubot-desktop-screenshot/kuflow_samples_kubot_desktop_screenshot/tasks-desktop-screenshot.py

import os

from PIL import ImageGrab


def take_a_desktop_screenshot_to_home():
    # Get the path to the user's home directory
    home_dir = os.path.expanduser("~")

    # Output file path
    image_path = os.path.join(home_dir, "desktop-screenshot.png")

    # Task screenshot
    screenshot = ImageGrab.grab()
    screenshot.save(image_path)

    print(f"Desktop screenshot has been written in {image_path}")


if __name__ == "__main__":
    take_a_desktop_screenshot_to_home()

The above function uses the Pillows library to take the screenshot, if you look at the "pyproject.toml" file you will see that it is included as a dependency. Now simply test the code, selecting Run or Debug Python file. If everything goes well, the capture should be created in your Home folder.

Building the KuBot package​

Now that we have a valid code, we will build the zip package that we will upload to KuFlow. To do this, we will first obtain a requirements.txt file that describes the necessary dependencies in the project. To do this we will execute:

poetry export --without-hashes --format=requirements.txt > requirements.txt

# NOTE: In future versions of Poetry you must first install the export plugin:
# $POETRY_HOME/bin/pip install --user poetry-plugin-export

The next step is to create our manifest, we create a kubot.yaml file with the following content. This file is already created in our example (kuflow-samples-kubot-desktop-screenshot/kubot.yaml), if you wish, edit it and leave only one operation (later we will add the other).

#
# kubot.yaml manifest
#

operations:
  - name: Desktop screenshot to User's Home
    shell: >-
      python kuflow_samples_kubot_desktop_screenshot/tasks-desktop-screenshot.py

tools:
  python: 3.11.4

post-install:
  - name: Install dependencies
    shell: pip install -r requirements.txt

Next we create the zip:

# Create a zip with the following structure
# ZIP_ROOT:
#	./rpa_estur
#	./requirements.txt
#	./kubot.yaml
zip -r kubot.zip kuflow_samples_kubot_desktop_screenshot requirements.txt kubot.yaml

Upload the KuBot package​

You can check the application reference to know how to do it. For our example you only need to create an assisted robot, upload the package and give it a name and a code.

  • Name: Desktop screenshot
  • Description: Take a screenshot of the user's desktop.
  • Code: DESKTOP_SCREENSHOT
  • Source type: Robot code file
  • Source code file: our package
  • Permissions: Groups > Users (available for all users in our organization)

Attach the robot to a task​

We will return to the definition of our process, select the task and edit it.

  • Assisted robot: We select "Desktop screenshot"
  • Assisted robot operation: We select "Desktop screenshot to User's Home"

We save and publish the changes, so they are available for instantiation.

Install the KuBot in the client​

We will open the KuBot Manager desktop application, log in with our KuFlow credentials and our robot should appear. We will click on install and wait a few minutes until the installation is complete. Once completed we are ready to receive jobs. If you close the application by pressing "X" the application will be minimized to your task tray so that you can continue going about your normal workday while listening to jobs for the robot.

Test the KuBot​

We are ready to test our robot by instantiating it in a task of our business process. To do this, we instantiate our process in the KuFlow cloud app.

Next, a human assisted task will be created for us:

Clicking on "Start" will launch the job request that KuBot Manager will listen to and perform the job. If everything has gone correctly, we enable the "Complete task" button, we complete and finish our process.

If on the user's computer, go to the user's home folder (usually something like C:\Users\username), we will see the screenshot created with the name "screenshot.png".

Congratulations! You have already developed your first KuBot.

Process the result​

Now we will complicate our KuBot a little more in order to upload the captured image to the task in the app. To do this, the first thing we must do is modify our structure data associated with the task to establish a file field. Read more about how to do this in the associated documentation.

Schema:

{
  "properties": {
    "file": {
      "type": "string",
      "format": "kuflow-file",
      "accept": ".png,.jpg,.jpeg",
      "title": "Screenshot file",
      "readOnly": true
    }
  }
}

UI Schema:

{
  "type": "Control",
  "scope": "#/properties/file"
}

The rest of the fields will be left as they are. Once the changes have been made, we save them.

To upload the image, we will use the rest client provided by the KuFlow SDK. You can see the dependency in the file pyproject.toml. To prove that multiple KuBots can coexist within the same package, we will create another file with our new code. You can see it here: kuflow-samples-kubot-desktop-screenshot/kuflow_samples_kubot_desktop_screenshot/tasks-desktop-screenshot-to-kuflow.py

Basically, a new method has been written that uploads the generated file; for this, the rest client for KuFlow has also been initialized. Explore the rest of the code to take a closer look at these parts. For now take a look at the upload code:

def _upload_file(task_id: str, doc_path: str) -> kf_models.TaskSaveJsonFormsValueDocumentResponseCommand:
    command = kf_models.TaskSaveJsonFormsValueDocumentRequestCommand(schema_path="#/properties/file")

    file_name = os.path.basename(doc_path)
    file = open(doc_path, "rb")
    content_type = _guess_content_type(doc_path)
    file = kf_models.Document(
        file_mame=file_name,
        content_type=content_type,
        file_content=file,
    )

    value = ROBOT_CONTEXT.kuFLow_client.task.actions_task_save_json_forms_value_document(task_id, file, command).value

    json_model = {"file": value}
    json_text = json.loads(json.dumps(json_model))

    command_save_json = kf_models.TaskSaveJsonFormsValueDataCommand(data=json_text)

    return ROBOT_CONTEXT.kuFLow_client.task.actions_task_save_json_forms_value_data(
        id=task_id, command=command_save_json
    )

A function has also been created to add log messages to the task. These messages help give feedback to the user if desired.

def _append_log_message(message: str, level: kf_models.LogLevel) -> kf_models.Task:
    task_id = os.environ.get(KuFlowEnvironmentVariablesConstants.KUFLOW_TASK_ID.value, None)

    log = kf_models.Log(message=message, level=level)
    task = ROBOT_CONTEXT.kuFLow_client.task.actions_task_append_log(task_id, log)

    return task
IMPORTANT

This code needs environment variables that inject, among other things, authentication tokens. KuBot Manager does it for us. You can read more here. To run the robot outside of KuBot Manager (while it is developing), you must provide these environment variables in your environment (usually through the IDE). Later we will see how.

The next step is to add this new robot to our manifest "kubot.yaml". As you can see, different KuBots can coexist in the same package if you wish.

#
# kubot.yaml manifest
#

operations:
  - name: Desktop screenshot to User's Home
    shell: >-
      python kuflow_samples_kubot_desktop_screenshot/tasks-desktop-screenshot.py

  - name: Desktop screenshot to KuFlow
    shell: >-
      python -m kuflow_samples_kubot_desktop_screenshot.tasks-desktop-screenshot-to-kuflow

tools:
  python: 3.11.4

post-install:
  - name: Install dependencies

With our new code and the manifest (we already had the dependencies before), we will create the package as we did before and update our robot in the KuFlow app. As we have created a new operation, we must go to the task definition and indicate the new operation that we want to execute, in our case we choose:

  • Assisted robot operation: Desktop screenshot to KuFlow

Make sure at this point to publish all pending changes to the process.

Later we will go to KuBot Manager. You will see an orange dot in the list indicating that a code update is available. We must install it before we can instantiate a new robot.

Well, we are now ready to re-instantiate a new process. We go to the KuFlow app and run the "Sample KuBot desktop screenshot" process. And finally, success! In our task we can now see the capture made.

Kuflow Logo
Legal
Copyright © 2024 KUFLOW, S.L.