Simple Tensorflow AI Decision

The Simple TensorFlow AI Decision plugin enables the integration of pre-trained TensorFlow models into process workflows for automated decision-making. This documentation outlines how to configure and utilize the plugin effectively.

Prerequisites

Before using the Simple TensorFlow AI Decision plugin, ensure the following items are prepared:

• Exported Frozen Model: Obtain the TensorFlow AI model in .pb format.
• Input Names and Data Types: List all input variables the model requires and their respective data types.
• Output Names and Data Types: Define the model's output variables and their data types.
• Dictionary File: If necessary, prepare a dictionary file in .csv format.

Simple TensorFlow AI 

To configure Simple Tensorflow AI, drag and drop the Route element onto the canvas in Process Builder, click it, and select Mapping.

When adding Simple TensorFlow AI in the field Plugin, you will see the following fields available for configuration:

• TensorFlow Session:
  • Add Session: Add a Session. See Add Session properties below.
  • Add Post Processing: 

    When hovering the mouse over the Add Post Processing, a list of Post Processing types will be available as shown below:

    • BeanShell Script - see Add Post Processing - BeanShell Script properties.
    • Image Detection Boxes - see Add Post Processing - Image Detection Boxes properties.
    • Values to Labels - see Add Post Processing - Detected Values to Labels properties.
    • Euclidean Distance - see Add Post Processing - Euclidean Distance properties.
    • Probabilities to Labels - see Add Post Processing - Probabilities to Labels properties.
• Rules:
  • Add Rule: Add a Rule. See Add Rule properties below.
  • Add Action: Adds an Action within the ELSE THEN statement. See Add Action properties below.

Add session

Add Session allows users to define a TensorFlow session for running AI models. Users can select the AI model file, delete sessions, and add inputs and outputs to process the model data within their workflows.

Fields to configure:

• Model: 
  • Choose File - Select an AI model to run
  • Clear - clear selected AI model
• Delete Session: Removes a defined session.
  
• Add Input: 

  When hovering the mouse over the Add Input, a list of Pre-Processing Input types will be available as shown below:

  • BeanShell Script - see Add Input - Beanshell Script properties.
  • Image - see Add Input - Image properties.
  • Boolean - see Add Input - Boolean properties.
  • Audio MelSpectogram - see Add Input - Audio Melspectogram properties.
  • File (Bytes) - see Add File(Bytes) - File Bytes properties.
  • Numbers - see Add Input - Numbers properties.
  • Text - see Add Input - Text properties.
• Add Output: See Add Output properties.

Add input

Add Input enables users to configure the inputs the TensorFlow model requires. Users can select from input types such as BeanShell Script, Image, Boolean, Audio MelSpectogram, File (Bytes), Numbers, and Text, ensuring the model receives the necessary data for accurate decision-making.

BeanShell Script

Users can define an input tensor and provide a Java script. This allows for custom pre-processing of input data before it is fed into the TensorFlow model, enhancing the flexibility and accuracy of AI-driven decisions.

When you select BeanShell Script in Add Input, you will see the following fields available for configuration:

• Input Name: Defined Input Tensor.
• Script: Script in Java. See Bean Shell Programming Guide for code examples.
• Delete Input: Removes a defined input.
  

Image

Users can define an input tensor for image data. Configuration options include specifying the file source, input tensor type, and image properties such as width, height, mean, and scale, ensuring proper image input handling for the TensorFlow model.

When you select Image in Add Input, you will see the following fields available for configuration:

• Input Name: Defined Input Tensor.
• Type:
  • Float
  • Double
  • Integer
  • UInt8
  • Long
• File Source:
  • Retrieve from URL
  • <list of all available Forms>
• URL or Image Upload Field ID:
  • URL input if Retrieve from URL was selected as the File Source.
  • Image Upload Field ID if a Form was selected as the File Source.
• Width: Width in Integer.
• Height: Height in Integer.
• Mean: Mean in Integer.
  
  Note: If it's not defined, set it to 1.
• Scale: Scale in Integer.
  
  Note: If it's not defined, set it to 1.
• Delete Input: Removes a defined input.
  

Boolean

Users can define an input tensor for Boolean data. This configuration allows users to specify true or false values, enabling the model to process logical conditions within the workflow.

When you select Boolean in Add Input, you will see the following fields available for configuration:

• Input Name: Defined Input Tensor.
• Type:
  • True
  • False
• Delete Input: Removes a defined input.
  

Audio MelSpectogram

Users can configure the input tensor for audio data. Options include specifying the file source, FFT size, overlap size, and frequency range, ensuring the model can accurately process audio inputs for decision-making.

When you select Audio MelSpectogram in Add Input, you will see the following fields available for configuration:

• Input Name: Defined Input Tensor.
• File Source: 
  • Retrieve from URL.
  • <list of all available Forms>
• URL or File Upload Field ID:
  • URL input if Retrieve from URL was selected as the File Source.
  • File Upload Field ID if a Form was selected as the File Source.
• Width: Width in Integer.
• Height: Height in Integer.
• FFT Size: FFT Size in Integer.
• Overlap Size: Overlap Size in Integer.
• Min Frequency (Hz): Minimum Frequency in (Hz).
• Max Frequency (Hz): Maximum Frequency in (Hz).
• Delete Input: Removes a defined input.
  

File (Bytes)

Users can define an input tensor for file data. Configuration includes specifying the file source and file upload field ID, allowing the TensorFlow model to process binary file data within the workflow.

When you select File (Bytes) in Add Input, you will see the following fields available for configuration:

• Input Name: Defined Input Tensor.
• File Source: 
  • Retrieve from URL
  • <list of all available Forms in this app>
• URL or File Upload Field ID:
• Delete Input: Removes a defined input.
  

Numbers

Users can define an input tensor for numerical data. Users can specify the type of numbers and input the values separated by a delimiter, ensuring the model receives the correct numerical inputs for processing.

When you select Numbers in Add Input, you will see the following fields available for configuration:

• Input Name: Defined Input Tensor.
• Type:
  • Float
  • Double
  • Integer
  • UInt8
  • Long
• Number Values (Separated by): separate number values by ;
• Delete Input: Removes a defined input.
  

Text

Users can define an input tensor for text data. Configuration options include specifying the text value, dictionary file, and maximum length, allowing the model to accurately process textual inputs for decision-making.

When you select Text in Add Input, you will see the following fields available for configuration:

• Input Name: Defined Input Tensor.
• Type: 
  • Float
  • Double
  • Integer
  • UInt8
  • Long
• Text Value: The text value inputted to be inserted into the Tensor. You can use a Hash Variable.
• Dictionary (JSON/CSV): Dictionary in JSON or CSV format.
• Max Length: Maximum Length 
• Leave Empty In Front: check to leave the front of the Value as Empty (or whatever value is based on the dictionary).
• Delete Input: Removes a defined input.
  

Add output

Add Output allows users to define the output tensor for the TensorFlow model. Users can specify the output name and assign it to a temporary variable, enabling further processing and integration of the model's results within the workflow.

When you select + Add Output in Outputs, you will see the following fields available for configuration:

• Output Name: Defined Output Tensor.
• Temporary Variable Name: The Output Tensor Value will be inserted into this user-defined Temporary Variable to later use during Post-Processing.
• Delete Output: Removes a defined input.
  

Add post processing

Add Post Processing enables users to configure post-processing steps for the TensorFlow model outputs. Options include BeanShell Script, Image Detection Boxes, Values to Labels, Euclidean Distance, and Probabilities to Labels, allowing users to refine and utilize the model's results effectively.

BeanShell Script

BeanShell Script allows users to define custom scripts to further process TensorFlow model outputs. Users can write Java code to manipulate the output data as needed, providing flexibility and control over post-processing steps.

When you select BeanShell Script in + Add Post Processing, you will see the following fields available for configuration:

• Post Processing Name: User-defined variable. It can be used as a Variable in the Rules.
• Script: 

  Script in Java. See  Bean Shell Programming Guide for code example.

  Injected variables available for consumption are:-

  • String Name
  • Map tfVariables
  • Map Variables
  • Map tempDataHolder
• Delete Post-Processing: Removes a defined Post-Processing.
  

Image detection boxes

Image Detection Boxes enable users to configure post-processing for image detection results. Users can specify font size, file source, output file details, and upload label and color files. This helps visualize detected objects within images by drawing bounding boxes and assigning labels.

When you select Image Detection Boxes in + Add Post Processing, you will see the following fields available for configuration:

• Post Processing Name: User-defined variable. Can be used as a Variable in the Rules.
• Font Size (pixel): The font size of detected labels in pixels.
• File Source: 
  • Retrieve from URL
  • <list of all available Forms in this app>
• URL or Image Upload Field ID:
  • URL input if Retrieve from URL was selected as the File Source.
  • File Upload Field ID if a Form was selected as the File Source.
• File Output Source:
  • Retrieve from URL.
  • <list of all available Forms in this app>.
• Output Image Upload Field ID:
  • URL input if Retrieve from URL was selected as the File Source.
  • File Upload Field ID if a Form was selected as the File Source.
• Output Image File Name: User-defined variable. Name of the output image file.
• Labels (TXT): In this field, upload a .txt file containing the labels of the detected classes.
• Colors (TXT): In this field, upload a .txt file containing the labels of the detected classes.
• Detected Classes: <List of classes detected in from output>.
• Number of Detected: The number of the classes detected in the input source.
• Detected Scores: <List of scores detected from output>.
• Detected Scores Threshold: Set the threshold of the detected scores (e.g 0.01).
• Detected Boxes: <List of detected boxes from output>.
• Boxes Type:
  • Top left Corner (x,y), width & height.
  • Top left Corner (y,x), width & height.
  • Top left Corner (x,y) & bottom right corner (x,y).
  • Top left Corner (y,x) & bottom right corner (y,x).

Detected values to labels

Detected Values to Labels allows users to map detected values from the TensorFlow model to human-readable labels. Users can specify a dictionary file and configure thresholds and unique value settings, facilitating a more straightforward interpretation of the model's output.

When you select Image Detection Boxes in + Add Post Processing, you will see the following fields available for configuration:

• Post Processing Name: User-defined variable. It can be used as a Variable in the Rules.
• Get Unique Only: Check to get unique values only.
• Labels (TXT): Dictionary file in .txt format
• Detected Classes: <List of classes detected from output>.
• Number of Detected: The number of the classes detected in the input source.
• Detected Scores: <List of scores detected from output>.
• Detected Scores Threshold: Set the threshold of the detected scores (e.g 0.01).
• Delete Post-Processing: Removes a defined Post-Processing.
  

Euclidean distance

Euclidean Distance enables users to calculate the Euclidean distance between vectors in the TensorFlow model's output. This can be useful for tasks like similarity detection. Users can define a post-processing name and specify temporary variables for the calculation.

When you select Euclidean distance in + Add Post Processing, you will see the following fields available for configuration:

• Post-Processing Name: User-defined variable. It can be used as a Variable in the Rules.
• Temporary Variable Name: List of Temporary Variables from Output.
• Temporary Variable Name: List of Temporary Variables from Output.
• Delete Post-Processing: Removes a defined Post-Processing.
  

Probabilities to labels

Probabilities to Labels allows users to convert probability scores from the TensorFlow model into labels. Users can set a threshold, choose to get the highest value only and upload a dictionary file for label mapping. This helps interpret probabilistic outputs as categorical labels.

When you select Probabilities to labels in + Add Post Processing, you will see the following fields available for configuration:

• Post Processing Name: User-defined variable. It can be used as a Variable in the Rules.
• Threshold: Set the Threshold value, e.g. 0.01.
• Get Highest Only: Check to get the highest value only.
• Labels (TXT): Dictionary file in .txt format.
• Temporary Variable Name: List of Temporary Variables from Output.
• Delete Post-Processing: Removes a defined Post-Processing.
  

Add rule

Add Rule allows you to define specific rules within a decision process. Users can toggle between equality and inequality conditions and specify how multiple conditions are evaluated using AND or OR logic. Additional options include adding groups of conditions, reordering rules, and removing rules as needed.

Fields to configure:

• Toggle between EQUAL and NOT EQUAL: Determines whether the condition checks for equality or inequality.
   
• Split Type: Specifies how multiple conditions within a rule are evaluated.
  
  • AND
  • OR
    
• Add Condition: Adds specific conditions within the IF statement.  Below, see Add Condition properties.
• Add Group: Adds a Group within the IF statement. Properties will be similar to Add Rule properties.
• Delete Rule: Removes a defined rule.
  
• Sort: Allows reordering of rules.
  
• Add Action: Adds an Action within the THEN statement.

Add condition

Add Condition enables the definition of conditions within an IF statement. Users can toggle between equality and inequality, specify the variable name, and select the type of operation (e.g., equal to, greater than). This feature allows for detailed conditional logic to be applied in decision-making processes.

When you select + Add condition, you will see the following fields available for configuration:

• Toggle between EQUAL and NOT EQUAL: Determines whether the condition checks for equality or inequality.
   
• Variable: The user inputs the variable name.
• Operation:
  • Equal To
  • Greater Than
  • Greater Than Or Equal To
  • Less Than
  • Less Than Or Equal To
  • Is True
  • Is False
  • Contains
  • List Contains
  • In
  • Regex Match
• Value: User to input the value of the Variable.
• Delete Condition: Removes a defined condition.
  

Add action

Add Action specifies the actions to be taken if the conditions in the IF statement are met. Users can choose between affecting transitions or workflow variables, define transition paths, and set workflow variable values. This feature ensures that appropriate actions are triggered based on the defined rules and conditions.

When you select + Add Action, you will see the following fields available for configuration:

• Type: Specifies whether the action affects transitions or workflow variables.
  • Transition: This will display a selection of all available transitions based on the route selected as part of the process flow.
    This field will be displayed if Transition is selected in the Type field.
  • Workflow Variable: Will display a selection of all available workflow variables based on the current Process.
    This field will be displayed if Workflow Variable is selected in the Type field.
• Value: Input field for specifying transition or workflow variable values.
  This field will be displayed if Workflow Variable is selected in the Type field.
• Delete Action: Removes a defined action.
  

Pre-trained AI model

Download the AI model, extract the zip, and add the 20180402-114759.pb file into the app's Resource in the App composer.

Credits

• Facenet TensorFlow: Provided the pre-trained AI model.
• VGGFace2: Contributed to the training dataset.