AI Builder in Power Automate By Ruchika Agarwal

 AI Builder

What is Power Platform?

Power Platform is a Microsoft low code and no code platform.

What does low code and no code platform means?

Its a solution and tools that you can use to build your own application, automations chatbots dashboards without having the knowledge on programming.

What are applications under Power Platform?

We have four big application under power platform.

1. Power BI :- Where we can build the dashboards, Work on all the data
2. PowerApps:- Here we can build mobile, desktop and wed Apps.
3. Power Automate:- Its all about automating the task.
4. Power Virtual Agent:- Which allows you to build the chatbots.

What is Horizontal capabilities?

1. Data Connectors:- This allows all this applications of power platform to connect to more that 800 services both Microsoft and non Microsoft services.
2. Dataverse:- This is our local data storage solution.
3. AI Builder:- This adds the AI to all these application and automations.

What is AI Builder?

• AI Builder sits within the power platform.

Form where does AI comes from?

• AI Builder comes from an existing feature set of Azure AI, Its just encapsulated for citizen developers.
• AI builder is build on the top of Azure AI making it non code.

What are AI Builder Capabilities?

We group them into four pillars which are mentioned below:-

1.) Documents 

• Process forms and structured documents
• Recognize text (OCR)
• Process invoices and receipts
• Process passports and US driver's licenses  
• Process business cards

2.) Text

• Categorize text 
• Extract entities
• Extract key phrases
• Translate
• Analyze sentiment

3.) Structured Data 

• Predict quantity  
• Predict outcomes

4.) Images

• Detect objects
• Classify images
• Detect language

What is Custom vs. pre-build models for documents?

• Text Recognizer :- 
• Basic OCR extracting printed or handwritten text from a document or Image. 
• The text recognizer model will return all the text including the hand written text and in more than 100 languages.
• Eg:- Extract and Identifier or a text pattern from a document. 
• Standard Models :-
• Models pretrained to extract data from standard document type.
• Ed:- Extract most common fields from invoices, Receipts, Business card, passports or Driving license.
• Form Processing :-
• Intelligent OCR. Train an AI Model to understand a unique layout structure.
• Teach the model and the model is gonna learn then its gonna extract and automate that information.
• Eg:- Used when existing prebuild model did not succeed to extract all fields. Also, to extract data from other type of document (Loyalty Card, Tax Form, Inventory Form).
  

What types of documents can we send to the form processing model when we use it? What types of documents can we send to the form processing model when we use it?

We can only provide the PDF and JPEG,PNG and T-files images.

Does the AI model have the ability to learn on its own?

As of now no, it works on feedback. You provide feedback to Microsoft they will retrain the model.

But the confidence score increases as we train more and more documents.

What is Starter kit Solution?

Document Automation is the starter kit in power automate.

What are the similarities and differences of AI builder and azure cognitive services?

Similarities:- AI Builder is build on top of Azure Cognitive Services So, you can expect the same quality same results and same performance the same scalability.

Differences:- API flexibility is not there in AI builder but we have that in  Azure Cognitive Services.

AI Builder is Platform-level toolset, simplicity of integration and Azure Cognitive Services are well-established APIs and a wide range of options. In simple words its No code and Pro code.

What are limitations of AI Model?

• We are unable to expand on the current model.
• We are unable to obtain Signature as an image.
• Multi-page table extraction is not possible. Tables that expand across multiple pages: For the time being, Microsoft is working on an experimental feature that allows you to extract the table across multiple pages.
• Dates extracted from existing models are only in text format, but we can now normalize the value and use that date format in our flow.

Integration in Power Automate

Use the pre-configured templates to create an end to end process.

• Choose how to use your model in power automate to quickly create a fully automated process.
• Choose from the pre-configured templates based on where you want to extract and send your data.

Documents:- 

• Intelligent document processing (IDP) refers to a set of business solutions that automate document processing using deep learning tools.
• IDP leverage deep learning AI model that is trained to understand the structure of the document and return the information you need to process.
• Through AI you can identify the fields, tables and checkboxes in a document.

How does it works?

Power Automate and the Power Platform provide a complete solution for large-scale document processing, including AI-powered data extraction, manual validation, process orchestration, pipeline monitoring, and other features. 

1. Receive Documents
2. Extract
3. Validate
4. Export
5. Monitor

Example 1:- To use the pre build Invoice Processing.

Step1:- Go to Power Automate and click on AI builder.

Step2:- Expand the AI builder.

Step3:- Select the pre-build model of invoice processing.

Step4:- Here we can do the testing how our flow will run and post that click on Use in Flow.

Step5:- Just connect to your official mail id and hit continue.

Step6:- Here we go, this is ready for us though the existing/ pre-build model.

Step7:- For testing just upload this reference invoice.

Step8:- As we have uploaded the above given image for testing it did the testing and our flow has ran successfully.

Step9:- We have received an email with the complete details.

The preceding example demonstrates how the pre-built AI can be used in our flow with just a few clicks and no code. This is how we can use a pre-built invoice processing model in Power Automate. 

Example 2:- This is an example of a Custom Model. Here, we will create our own model, train it, manage it, publish it, use it, and share it. We will train our model for Indian Post tracking receipt were we will be tracking the following :- 1.) Date, 2.) Tracking no, 3.) From, 4.) To, 5.) Destination Pin Code, 6.) Starting Pin Code, 7.) Weight, and 8.) Amount.

Step 1:- Go to Power Automate -> AI Builder --> Explore --> Select Document Processing (Custom Model)  

Step 2:- Post Selecting custom model we will this window click on Get Started

Step 3:- We have five steps to train our model.

• Choose document type
• Choose information to extract
• Add collections of documents
• Tag documents
• Model Summary

Step 4:- First choose the document type:-

Step 5:- As we have Structure documents select that and hit on next.

Step 6:- Second is choose the information to extract. 

Step 7:- Here we can select multiple fields, checkboxes and tables.

Step 8:- In this example we require the below mentioned fileds.

Step 9:- Now add the set of documents to be trained.

Step 10:- We have to add minimum five documents to train.

Step 11:- Now Tag the documents. 

Step 12:- here we will train our model which information belongs to which filed. Here all the set of collection will be tagged.

Step 13:- At last see the model summary.

Step 14:- We will see here our model summary.

Step 15:- Now train the model

Step 16:- Now model is getting trained.

Step 17:- Publish the model.

Step 18:- Now under AI Builder --> Model --> we can see the recent model which we have trained.

Step 19:- Now we will use this trained model in Flow--> Use Model

Step 20:- We have used the recent trained model in our flow where manually we will be uploading the tracking receipt and will be getting the out in an excel.

Step 21:- Now text our flow.

Step 22:- Upload this receipt as an test image and test the flow.

Step 23:- Our flow ran successfully. now we will be checking whether the desired output has received or not. 

Step 24:- Here we go we can see the Output as expected and we can further use this in our Apps or anywhere its required . we can share the same with our team.

Summary:- We have trained the model published it and used it around the flows and apps.

Example 3:- Translate text into another language using AI Builder on button click

Step1:- Use the pre build model for converting the text into another language.

Step2:- Text the flow by providing the input and into which language code you want to convert it.

Step3:- Our flow ran successfully. 

Step4:- We received the email with the final output.  

Example 4:- Extract text from images or PDF documents using AI Builder Text Recognition

Step1:- Here we will use the pre build model that is recognize text in an image in the flow.

Step2:- Now we will test the flow.

Step3:-  This is a JPG Image that we are uploading for testing.

Step4:-  Here our flow ran successfully. 

Step5:- Here we go...we got an email where we got text from an uploaded image. 

Example 5:-Detect Sentiment of a text.

Step1:- We will use the pre build model that is analyze positive or negative sentiment in text.

Step2:- Text the flow my entering the input text.

Input Text: -

•  I am happy today
• I am sad today
• I am okay

Step3:- Here we go it shows the confidence score and the final sentiment.

These are the five most important examples we've seen. We can try multiple in-build or custom models based on our needs.

Conclusion of AI builder:-

AI Builder enables you to accelerate and improve your business process capability by analyzing, extracting data, and predicting future outcomes that benefit your business using predefined models or creating your own custom models. AI developers are improving the efficiency and effectiveness of power platforms.

Thank you for your time:)