UnitV2 recognition service

Driver Installation

Download the corresponding SR9900 driver according to the operating system used.

Windows10

Extract the driver compressed package to the desktop path -> Enter the device manager and select the currently unrecognized device (named with SR9900) -> Right-click and select Custom Update -> Select the path where the compressed package is decompressed -> Click OK and wait for the update carry out.

MacOS

Unzip the driver package -> double-click to open the SR9900_v1.x.pkg file -> follow the prompts and click Next to install. (The compressed package contains a detailed version of the driver installation tutorial pdf)

  • After the installation is complete, if the network card cannot be enabled normally, you can open the terminal and use the command below to re-enable the network card.
sudo ifconfig en10 down
sudo ifconfig en10 up

Connect the device

After connecting the USB for power supply, UnitV2 will start automatically, the power indicator will be red and white, and it will go out after the start is complete. UnitV2 internally integrates the basic AI recognition application developed by M5Stack, and has built-in multiple recognition functions (such as face recognition, object tracking and other common functions), which can quickly help users build AI recognition applications. Through the following two connection methods, the PC/mobile terminal can access the domain name unitv2.py or IP: 10.254.239.1 through the browser to access the preview webpage through the recognition function. During the identification process, UnitV2 will continuously output identification sample data (JSON format, UART: 115200bps 8N1) through the serial port (HY2.0-4P interface at the bottom)

Note: The built-in recognition service has some compatibility issues on the Safari browser. It is recommended to use the Chrome browser to access.

-Ethernet Mode Connection: UnitV2 has a built-in wired network card. When you connect to a PC through the TypeC interface, it will automatically establish a network connection with UnitV2.

-AP mode connection: After UnitV2 is started, the AP hotspot (SSID: M5UV2_XXX: PWD:12345678) will be turned on by default, and the user can establish a network connection with UnitV2 through Wi-Fi access.

Built-in function

Function switch

Switch between different recognition functions by clicking the navigation bar of the function page or sending JSON commands through Serial Port communication. Note: Except at the end of the sent command string, no line breaks are allowed in other positions.

//The value of the function key can be specified as any of the following functions
Audio FFT
Code Detector
Face Detector
Lane Line Tracker
Motion Tracker
Shape Matching
Camera Stream
Online Classifier
Color Tracker
Face Recognition
Target Tracker
Shape Detector
Object Recognition
//Please note that args must be a list.
{
    "function":"Object Recognition",
    "args":[
        "yolo_20class"
    ]
} 

Data format of switching function status response


//If the function switch is successful, you will receive a reply
{
    "msg":"function switched to Object Recognition."
}

//If the specified function does not exist, you will receive a reply
{
    "error":"function Object Recognition not exist"
} 

//If the function switch fails, you will receive a reply
{
    "error":"invalid function."
}

1. Camera Stream

1.1 Description

480P real-time video preview.

480P real-time video preview.

1.2 Serial Operation

Switch function to Camera Stream

{
    "function": "Camera Stream",
    "args": ""
}

2. Code Detector

2.1 Description

Identify the QR code on the screen, and return the coordinates and content of the QR code.

2.2 Web page operation

2.2 Serial Operation

Please switch the function to Code Detector

{
    "function": "Code Detector",
    "args": ""
}

2.3 Sample output

{
    "running":"Code Detector",
    "num":2, // Number of QR codes
    "code":[
        {
            "prob": 0.987152, // confidence rate
            "x":10, // 0 ~ 640
            "y":10, // 0 ~ 480
            "w":30,
            "h":30, // the bounding box of the code
            "type":"QR/DM/Maxi",  // include "Background", "QR/DM/Maxi", "SmallProgramCode", "PDF-417", "EAN", "Unknown"
            "content":"m5stack"
        },
        {
            "prob": 0.987152, // confidence rate
            "x":10,
            "y":10,
            "w":30,
            "h":30, // the bounding box of the code
            "type":"QR/DM/Maxi",  // include "Background", "QR/DM/Maxi", "SmallProgramCode", "PDF-417", "EAN", "Unknown"
            "content":"m5stack"
        }
    ]
}

3. Object Recongnition

3.1 Description

Target detection based on YOLO Fastest and NanoDet. Support V-Training.

3.2 Web page operation

  1. Click Add to upload the model file. tar format, please refer to the tutorial for training custom models UnitV2 V-Training
  2. After selecting the model, click Run to run the specified model.(Built-in models: nanodet_80class, yolo_20classs can be used directly)

3.3 Serial Operation

Please switch the function to Object Recognition

//Select the parameter "yolo_20class" to switch to this function
{
    "function": "Code Detector",
    "args": ["yolo_20class"]
}
//Select the parameter "nanodet_80class" to switch to this function
{
    "function": "Code Detector",
    "args": ["nanodet_80class"]
}

Recognized objects supported by the built-in model


nanodet_80class: [
        "person","bicycle","car","motorbike","aeroplane","bus","train","truck","boat","traffic light",
        "fire hydrant","stop sign","parking meter","bench","bird","cat","dog","horse","sheep","cow",
        "elephant","bear","zebra","giraffe","backpack","umbrella","handbag","tie","suitcase","frisbee",
        "skis","snowboard","sports ball","kite","baseball bat","baseball glove","skateboard","surfboard",
        "tennis racket","bottle","wine glass","cup","fork","knife","spoon","bowl","banana","apple",
        "sandwich","orange","broccoli","carrot","hot dog","pizza","donut","cake","chair","sofa","pottedplant",
        "bed","diningtable","toilet","tvmonitor","laptop","mouse","remote","keyboard","cell phone","microwave",
        "oven","toaster","sink","refrigerator","book","clock","vase","scissors","teddy bear","hair drier","toothbrush"
]

3.4 Sample output

{
    "num": 1,
    "obj": [
        {
            "prob": 0.938137174,
            "x": 179,
            "y": 186,
            "w": 330,
            "h": 273,
            "type": "person"
        }
    ],
    "running": "Object Recognition"
}

4. Color Tracker

4.1 Description

Detect the specified color area and return the coordinates of the color area.

4.2 Web page operation

You can directly adjust the LAB threshold slider to filter out the background and get the color area of interest. You can also directly frame the color area of interest on the screen. The system will automatically calculate the color with the largest proportion of the target area and filter out the background. You can further adjust the slider on the basis of the calculation to achieve a better filtering effect. Click the "To Mask Mode" button to switch to Mask mode, in which you can directly see the filtering effect. Click the "To RGB Mode" button again to switch back to RGB mode.

  • About CIELAB color space
  • The LAB threshold is mapped to 0~255.
  • The L in LAB stands for brightness. Usually, this threshold is not set (0~255), which means that the influence of brightness is ignored. But this will cause the algorithm to be unable to distinguish between black and white, please pay attention.
  • Algorithm will only return the largest goal

4.3 Serial Operation

Before performing the following operations, please switch the function to Color Tracker

{
    "function": "Color Tracker",
    "args": ""
}

4.3.1 Specify LAB threshold

  • Send
    // * LAB threshold is mapped to 0~255
    {
        "config":"Color Tracker",
        "l_min":0// 0 ~ 255
        "l_max":0// 0 ~ 255
        "a_min":0// 0 ~ 255
        "a_max":0// 0 ~ 255
        "b_min":0// 0 ~ 255
        "b_max":0 // 0 ~ 255
  • Receive
    {
        "running":"Color Tracker",
        "msg":"Data updated."
    }
    

4.3.2 Specify ROI (automatically calculate threshold)

  • Send
    {
    {
        "config":"Color Tracker",
        "x":0// 0 ~ 640
        "y":0// 0 ~ 480
        "w":30,
        "h":30,
    }
    
  • Receive
    // * va and vb refer to the degree of color dispersion in the ROI. If the dispersion is higher, the tracking effect will be poor.
    {
        "running":"Color Tracker",
        "a_cal":0.0,
        "b_cal":0.0//Calculate the threshold
        "va":0.0,
        "vb":0.0//Color dispersion rate
        "l_min":0//Fixed value 0
        "l_max":255//Fixed value 255
        "a_min":0// a_cal - (10 + (int)(va / 2.0f))
        "a_max":0// a_cal + (10 + (int)(va / 2.0f))
        "b_min":0// b_cal - (10 + (int)(vb / 2.0f))
        "b_max":0  // b_cal + (10 + (int)(vb / 2.0f))

4.4 Sample output

{
    "running":"Color Tracker",
    "cx"0// Center X coordinate
    "cy"0// Center Y coordinate
    "r"0//radius
    "mx"0// moment x position
    "my"0 // moment y position
}

5. LaneLine Tracker

5.1 Description

Detect the road line in the screen, fit it into a straight line, and return the straight line angle and coordinates.

5.2 Web page operation

You can directly adjust the LAB threshold slider to filter out the background and get the color area of interest. You can also directly frame the color area of interest on the screen. The system will automatically calculate the color with the largest proportion of the target area and filter out the background. You can further adjust the slider on the basis of the calculation to achieve a better filtering effect. Click the "To Mask Mode" button to switch to Mask mode, in which you can directly see the filtering effect. Click the "To RGB Mode" button again to switch back to RGB mode.

  • About CIELAB color space
  • The LAB threshold is mapped to 0~255.
  • The L in LAB stands for brightness. Usually, this threshold is not set (0~255), which means that the influence of brightness is ignored. But this will cause the algorithm to be unable to distinguish between black and white, please pay attention.

5.3 Serial Operation

Before performing the following operations, please switch the function to LaneLine Tracker

{
    "function": "Lane Line Tracker",
    "args": ""
}

5.3.1 Specify LAB threshold

  • Send
     // * LAB threshold is mapped to 0~255
    {
        "config":"Lane Line Tracker",
        "l_min":0// 0 ~ 255
        "l_max":0// 0 ~ 255
        "a_min":0// 0 ~ 255
        "a_max":0// 0 ~ 255
        "b_min":0// 0 ~ 255
        "b_max":0// 0 ~ 255
    }
    
  • Receive
    {
        "running":"Lane Line Tracker",
        "msg":"Data updated."
    }
    

5.3.2 Specify ROI (automatically calculate threshold)

  • Send
    {
        "config":"Lane Line Tracker",
        "x":0// 0 ~ 640
        "y":0// 0 ~ 480
        "w":30,
        "h":30,
    }
    
  • Receive
    //* va and vb refer to the degree of color dispersion in the ROI. If the dispersion is higher, the segmentation effect will be poor.
    {
        "running":"Lane Line Tracker",
        "a_cal":0.0,
        "b_cal":0.0//Calculate the threshold
        "va":0.0,
        "vb":0.0// Color dispersion rate
        "l_min":0//Fixed value 0
        "l_max":255//Fixed value 255
        "a_min":0// a_cal - (10 + (int)(va / 2.0f))
        "a_max":0// a_cal + (10 + (int)(va / 2.0f))
        "b_min":0// b_cal - (10 + (int)(vb / 2.0f))
        "b_max":0  // b_cal + (10 + (int)(vb / 2.0f))

5.4Sample output

{
    "running":"Lane Line Tracker",
    "x":0,
    "y":0//Base point of fitted line
    "k":0  //Slope of fitted line

6. Target Tracker

6.1 Description

Select the target on the screen and track it, using the MOSSE algorithm.

6.2 Web page operation

Just frame the target of interest on the screen.

6.3 Serial Operation

Before performing the following operations, please switch the function to Target Tracker

{
    "function": "Target Tracker",
    "args": ""
}

6.4 Sample output

{
    "running":"Target Tracker",
    "x":0,//0~640
    "y":0,//0~480
    "w":0,
    "h":0

7. Motion Tracker

7.1 Description

Detect and track the moving target, return the coordinates and angle of the target.

7.2 Web page operation

Click the'Set as background' button to confirm the background. The algorithm can adapt to a slowly changing background.

7.3 Serial Operation

Before performing the following operations, please switch the function to Motion Tracker

{
    "function": "Motion Tracker",
    "args": ""
}

7.3.1 Determine the background

  • Send

    // Sending this instruction will determine the background
    {
        "config":"Motion Tracker",
        "operation":"update"
  • Receive

    {
        "running":"Motion Tracker",
        "msg":"Background updated."
    }
    

7.4 Sample output

{
    "running":"Motion Tracker",
    "num":2,
    "roi":[
        {
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "angle":0.0,
            "area":0
        },
        {
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "angle":0.0,
            "area":0
        }
    ]
} 

8. Online Classifier

8.1 Description

This function can train and classify the objects in the green target frame in real time, and the feature values obtained from training can be saved on the device for next use.

8.2 Web page operation

  1. Click the'Reset' button to clear the existing categories and enter the training mode.
  2. Click the'+' button to create a new category.
  3. Select the category you want to train.
  4. Place the object to be trained in the green target box.
  5. Click the'Train' button to complete a training session.
  6. Change the angle of the object and repeat the training until you think the desired effect is achieved.
  7. Click the'save&run' button to save the training results on the device, and exit the training mode for object recognition and classification.

8.3 Serial Operation

Before performing the following operations, please switch the function to Online Classifier


{
    "function": "Online Classifier",
    "args": ""
}

8.3.1 Train

  • Send

    //This instruction will make the device enter the training mode, and extract the features once and store them in the specified category. If class_id does not exist, this class will be created.
    {
        "config":"Online Classifier",
        "operation":"train",
        "class_id":1// Integer (0 ~ N),ID of the class
        "class":"class_1" // String,the name of the class
    }
    
  • Receive

    {
        "running":"Online Classifier",
        "msg":"Training [class name] [num of training] times"
    }
    

8.3.2 Save&Run

  • Send

    {
        "config":"Online Classifier",
        "operation":"saverun",
    }
    
  • Receive

    {
        "running":"Online Classifier",
        "msg":"Save and run."
    }
    

8.3.3 Reset

  • Send

    //This command will make the device enter training mode and clear all categories.
    {
        "config":"Online Classifier",
        "operation":"reset",
    } 
    
  • Receive

    {
        "running":"Online Classifier",
        "msg":" Please take a picture."
    }
    

8.4 Sample output

{
    "running":"Online Classifier",
    "class_num":2//Number of recognized classes
    "best_match":"class_1"//Best match class
    "best_score":0.83838//Best match score
    "class":[ //Score of each class
        {
            "name":"class_1",
            "score":0.83838
        },
        {
            "name":"class_2",
            "score":0.66244
        }
    ]
}

9. Face Recognition

9.1 Description

Detect and recognize human faces.

9.2 Web page operation

  1. Click the Reset button to clear all existing faces.
  2. Click the + button to create a new face.
  3. Select the face you want to train.
  4. Look at the camera and make sure the face to be trained is in the right place.
  5. Click the Train button.
  6. During training, when the bounding box is yellow, it means training. At this time, you can slowly rotate your head to sample different angles to achieve better recognition results.
  7. If the bounding box turns red, it means that the target has been lost, usually because the face has changed too much. Please adjust the position of the face until the face is found again.
  8. When you think that the desired effect is reached, click Stop. Now the device can recognize the face.
  9. Click the Save button to save the characteristic data to the device for next use.

9.3 Serial Operation

Before performing the following operations, please switch the function to Face Recognition

{
    "function": "Face Recognition",
    "args": ""
}

9.3.1 Train

  • Send

    //To create a new face, please provide face_id (0 ~ N) in order.
    {
        "config":"Face Recognition",
        "operation":"train",
        "face_id":1// Integer (0 ~ N),Face ID
        "name":"tom" // String,The name of the face
    }
    //For example, there are currently 3 faces (0~2). To create a new face, you need to specify the id as 3.
    
  • Receive (Success)

    {
        "running":" Face Recognition ",
        "msg":"Training tom" // Training name
    }
    
  • Receive (Error)

    {
        "running":"Face Recognition",
        "msg":"Invalid face id"
    }
    

9.3.2 Stop Train

  • Send

    {
        "config":" Face Recognition ",
        "operation":" stoptrain",
    }
    
  • Receive

    {
        "running":"Face Recognition",
        "msg":"Exit training mode."
    }
    

9.3.3 Save&Run

  • Send

    {
        "config":" Face Recognition ",
        "operation":"saverun",
    }
    
  • Receive

    {
        "running":"Face Recognition",
        "msg":"Faces saved."
    }
    

9.3.4 Reset

  • Send

    //This command will delete all faces.
    {
        "config":"Face Recognition",
        "operation":"reset",
    }
    
  • Receive

    {
        "running":"Face Recognition",
        "msg":"Reset success"
    }
    

9.4Sample output

9.4.1 Training Mode

{
    "running":"Face Recognition",
    "status":"training"// training or missing
    "x":0,
    "y":0,
    "w":0,
    "h":0//Facial recognition bounding box
    "prob":0//Detection Confidence Rate
    "name":0,
}

9.4.2 Normal Mode (Match score>0.5)

{
    "running":"Face Recognition",
    "num":1//The number of faces recognized
    "face":[
        {
            "x":0// 0 ~ 320
            "y":0// 0 ~ 240
            "w":30,
            "h":30//Facial recognition bounding box
            "prob":0//Detection Confidence Rate
            "match_prob":0.8//Match confidence rate
            "name""tom",
            "mark":[ // landmarks
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
            ]
        },
    ]
} 

9.4.3 Normal Mode (Match score<=0.5)

{
    "running":"Face Recognition",
    "num":1//The number of faces recognized
    "face":[
        {
            "x":0// 0 ~ 320
            "y":0// 0 ~ 240
            "w":30,
            "h":30//Facial recognition bounding box
            "prob":0//Confidence rate
            "name""unidentified",
            "mark":[ // landmarks
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
            ]
        },
    ]
} 

10. Face Detector

10.1 Description

Detect the face in the picture and give a 5-point landmark.

10.2 Web page operation

10.3 Serial Operation

Before performing the following operations, please switch the function to Face Detector


{
    "function": "Face Detector",
    "args": ""
}

10.4Sample output

{
    "running":"Face Detector",
    "num":1//The number of faces recognized
    "face":[
        {
            "x":0,
            "y":0,
            "w":30,
            "h":30//Facial recognition bounding box
            "prob":0// Confidence rate
            "mark":[ // landmark
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                },
                {
                    "x":0,
                    "y":0
                }
            ]
        }
    ]
}

11. Shape Detector

11.1 Description

Detect the shape in the screen and calculate its area. Able to recognize squares, rectangles, triangles, pentagons, and circles.

11.2 Web page operation

Click the'Set as background' button to confirm the background. The algorithm can adapt to a slowly changing background.

11.3 Serial Operation

Before performing the following operations, please switch the function to Shape Detector

{
    "function": "Shape Detector",
    "args": ""
}
  • Send

    // Sending this instruction will determine the background
    {
        "config":"Shape Detector",
        "operation":"update"
  • Receive

    {
        "running":"Shape Detector",
        "msg":"Background updated."
    }
    

    11.4Sample output

{
    "running":"Shape Detector",
    "num":2,
    "shape":[
        {
            "name":"Rectangle"// "unidentified", "triangle", "square", "rectangle", "pentagon", "circle"
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "angle":0.0//Can be used when the shape is square or rectangular
            "area":0
        },
        {
            "name":"Rectangle"// "unidentified", "triangle", "square", "rectangle", "pentagon", "circle"
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "angle":0.0//Can be used when the shape is square or rectangular
            "area":0
        }
    ]
} 

12. Shape Matching

12.1 Description

Match the given arbitrary shape (but the shape preferably does not contain curves), the uploaded shape will be converted into characteristic data and saved on the device for next use.

12.2 Web page operation

  1. Click the add button to add a shape. You need to upload a shape template picture as shown in the figure below (png format, the shape is black and the background is white), and the file name will be the name of the shape.
  2. Click the reset button to clear all the uploaded shapes.
  3. Click the'Set as background' button to confirm the background. The algorithm can adapt to a slowly changing background.

12.3 Serial Operation

To be developed, not yet supported.

12.4Sample output

// The shape returned here is the file name of the uploaded template. Please note that if the confidence rate is lower than 30%, it will be marked as unidentified.
{
    "running":"Shape Matching",
    "num":2,
    "shape":[
        {
            "name":"arrow"//The name of your custom shape cannot be recognized when the confidence rate is less than 30
            "max_score":83//Confidence rate score, if the shape is unknown, there is no
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "area":0
        },
        {
            "name":"unidentified"// The name of your custom shape cannot be recognized when the confidence score is less than 30
            "x":0,
            "y":0,
            "w":0,
            "h":0,
            "area":0
        },
    ]
} 

13. Audio FFT

13.1 Description

Capture audio through the microphone on the device, perform real-time FFT (Fast Fourier Transform) and draw a time-frequency diagram. The green chart below is the RMS of the audio, indicating the current loudness.

  • The cutoff frequency of the microphone response is about 10KHz.

13.2 Web page operation

13.3 Serial Operation

None

13.4Sample output

None

Serial read

During the identification process, UnitV2 will continuously output identification sample data (JSON format, UART: 115200bps 8N1) through the serial port (HY2.0-4P interface at the bottom). Below are the case programs for reading recognition results under different platforms.

Arduino

The JSON string can be parsed using the ArduinoJson library.


void setup() {

  Serial.begin(115200);
  Serial2.begin(115200, SERIAL_8N1, 16, 17);

}

void loop() {

 if(Serial2.available()) {
   String recvStr = Serial2.readStringUntil('/n');
   if(recvStr[0] == '{'){
     Serial.print(recvStr);
   }
 }
  
}

Micropython


import machine
import json

uart1 = machine.UART(1, tx=16, rx=17)
uart1.init(115200, bits=8, parity=None, stop=1)

PROTOCOL_START = b'{'[0]

while True:
  if uart1.any():
    data = uart1.readline()
    if data[0] == PROTOCOL_START:
        json_data = json.loads(data)

Use python to call model files


from json.decoder import JSONDecodeError
import subprocess
import json
import base64

reconizer = subprocess.Popen(['/home/m5stack/payload/bin/object_recognition', '/home/m5stack/payload/uploads/models/nanodet_80class'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

reconizer.stdin.write("_{/"stream/":1}/r/n".encode('utf-8'))
reconizer.stdin.flush()

img = b''

while 1:
    doc = json.loads(reconizer.stdout.readline().decode('utf-8'))
    print(doc)
    if 'img' in doc:
        byte_data = base64.b64decode(doc["img"])
        img = bytes(byte_data)
    elif 'num' in doc:
        for obj in doc['obj']:
            print(obj)
On This Page