Download the appropriate SR9900 driver according to the operating system used.
Extract the driver zip package to the desktop path -> Go to the device manager and select the currently unrecognized device (with the name of USB 10/100 LAN
or with the character SR9900
) -> Right-click and select Customize Update -> Select the path where the zip package is extracted -> Click on Confirm and wait for the update to complete.
Unzip the driver zip package -> double click to open the SR9900_v1.x.pkg file -> click Next to install according to the prompts. (The package contains a detailed version of the driver installation tutorial pdf)
sudo ifconfig en10 down
sudo ifconfig en10 up
After connecting to USB for power supply, UnitV2 will start up automatically, the power indicator will show red and white color, and turn off when the startup is completed.UnitV2 integrates the basic Ai recognition application developed by M5Stack internally, and has built-in various recognition functions (e.g. face recognition, object tracking, and other commonly used functions), which can help users to build an Ai recognition application quickly. Through the following two connections, PC/mobile users can access the preview web page of the recognition function through the browser to the domain name unitv2.py
or IP:10.254.239.1
. During the identification process, UnitV2 will continuously output the identification sample data (in JSON format, UART: 115200bps 8N1
) through the serial port (HY2.0-4P connector at the bottom).
Safari
browser, we recommend Chrome
for access.Ethernet Mode Connection
: UnitV2 has a built-in wired network card, when you connect your PC through the TypeC port, it will automatically establish a network connection with UnitV2.
AP Mode Connection
: When UnitV2 starts up, it will turn on AP Hotspot(SSID: M5UV2_XXX: PWD:12345678)
by default, users can establish network connection with UnitV2 through WiFi access.
During the identification process, UnitV2 will continuously output the identification sample data (JSON format, UART: 115200bps 8N1
) through the serial port (bottom HY2.0-4P interface).
Toggle between different recognition functions by clicking on the navigation bar
of the function page or by sending a JSON command through the device's Serial interface
. Note: It is not allowed to insert line breaks in the command string except at the end.
Audio FFT
Code Detector
Face Detector
Lane Line Tracker
Motion Tracker
Shape Matching
Camera Stream
Online Classifier
Color Tracker
Face Recognition
Target Tracker
Shape Detector
Object Recognition
{
"function":"Object Recognition",
"args":[
"yolo_20class"
]
}
{
"msg":"function switched to Object Recognition."
}
{
"error":"function Object Recognition not exist"
}
{
"error":"invalid function."
}
480P real-time video preview
{
"function": "Camera Stream",
"args": ""
}
Recognize the QR code in the screen and return the coordinates and content of the QR code.
{
"function": "Code Detector",
"args": ""
}
{
"running":"Code Detector",
"num":2, // Number of QR codes
"code":[
{
"prob": 0.987152, // confidence level (math.)
"x":10, // 0 ~ 640
"y":10, // 0 ~ 480
"w":30,
"h":30, // Bounding boxes for QR codes
"type":"QR/DM/Maxi", // include "Background", "QR/DM/Maxi", "SmallProgramCode", "PDF-417", "EAN", "Unknown"
"content":"m5stack"
},
{
"prob": 0.987152, // confidence level (math.)
"x":10,
"y":10,
"w":30,
"h":30, // Bounding boxes for QR codes
"type":"QR/DM/Maxi", // include "Background", "QR/DM/Maxi", "SmallProgramCode", "PDF-417", "EAN", "Unknown"
"content":"m5stack"
}
]
}
Target detection based on YOLO Fastest and NanoDet. Support V-Training.
//Select the parameter "nanodet_80class" to switch to this function.
{
"function": "Object Recognition",
"args": ["nanodet_80class"]
}
By default, the mirror has built-in nanodet_80class, yolo_20classes models, which can be used directly, and the following objects are supported by the model
yolo_20class: [
"aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog",
"horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"
]
nanodet_80class: [
"person","bicycle","car","motorbike","aeroplane","bus","train","truck","boat","traffic light",
"fire hydrant","stop sign","parking meter","bench","bird","cat","dog","horse","sheep","cow",
"elephant","bear","zebra","giraffe","backpack","umbrella","handbag","tie","suitcase","frisbee",
"skis","snowboard","sports ball","kite","baseball bat","baseball glove","skateboard","surfboard",
"tennis racket","bottle","wine glass","cup","fork","knife","spoon","bowl","banana","apple",
"sandwich","orange","broccoli","carrot","hot dog","pizza","donut","cake","chair","sofa","pottedplant",
"bed","diningtable","toilet","tvmonitor","laptop","mouse","remote","keyboard","cell phone","microwave",
"oven","toaster","sink","refrigerator","book","clock","vase","scissors","teddy bear","hair drier","toothbrush"
]
{
"num": 1,
"obj": [
{
"prob": 0.938137174,
"x": 179,
"y": 186,
"w": 330,
"h": 273,
"type": "person"
}
],
"running": "Object Recognition"
}
Detect the specified color region and return the coordinates of the color region.
You can directly adjust the LAB threshold slider to filter out the background and get the color area of interest. You can also directly frame the color area of interest in the screen, the system will automatically calculate the target area of the largest proportion of the color and filter out the background, you can further adjust the slider bar on the basis of the calculation to achieve better filtering results. Click "To Mask Mode" button will switch to Mask mode, in this mode you can directly see the filtering effect. Clicking the "To RGB Mode" button will switch back to RGB mode.
About CIELAB Color Space
- LAB Threshold is mapped to 0255 - The L in LAB stands for Luminance, usually this threshold (0255) is not set to ignore the effect of luminance. However, this will cause the algorithm to not be able to distinguish between black and white, please note. - The algorithm will only return the largest target
{
"function": "Color Tracker",
"args": ""
}
// * LAB thresholds are mapped to 0~255
{
"config":"Color Tracker",
"l_min":0, // 0 ~ 255
"l_max":0, // 0 ~ 255
"a_min":0, // 0 ~ 255
"a_max":0, // 0 ~ 255
"b_min":0, // 0 ~ 255
"b_max":0 // 0 ~ 255
}
{
"running":"Color Tracker",
"msg":"Data updated."
}
{
"config":"Color Tracker",
"x":0, // 0 ~ 640
"y":0, // 0 ~ 480
"w":30,
"h":30,
}
// * va and vb refers to the degree of color dispersion within the ROI, if the dispersion is high, the tracking effect is poor.
{
"running":"Color Tracker",
"a_cal":0.0,
"b_cal":0.0, // Calculation Threshold
"va":0.0,
"vb":0.0, // Color dispersion rate
"l_min":0, // Fixed value 0
"l_max":255, // Fixed value 255
"a_min":0, // a_cal - (10 + (int)(va / 2.0f))
"a_max":0, // a_cal + (10 + (int)(va / 2.0f))
"b_min":0, // b_cal - (10 + (int)(vb / 2.0f))
"b_max":0 // b_cal + (10 + (int)(vb / 2.0f))
}
{
"running":"Color Tracker",
"cx": 0, // Center X-axis coordinates
"cy": 0, // Center Y-axis coordinates
"r": 0, // radius
"mx": 0, // moment x position
"my": 0 // moment y position
}
Detect the road line in the screen, fit it to a straight line, and return the angle and coordinates of the line.
{
"function": "Lane Line Tracker",
"args": ""
}
"config":Lane Line Tracker"
.{
"running":"Lane Line Tracker",
"x":0,
"y":0, // Base point of the fitted line
"k":0 // Slope of the fitted line
}
The MOSSE algorithm is used to select a target on the screen and track it. Tracking is done by simply framing the target of interest on the screen.
{
"function": "Target Tracker",
"args": ""
}
{
"running":"Target Tracker",
"x":0,//0~640
"y":0,//0~480
"w":0,
"h":0
}
Detects and tracks a moving target, returning the target's coordinates and angle. Click the 'Set as background' button to determine the background. The algorithm can be adapted to slowly changing backgrounds.
{
"function": "Motion Tracker",
"args": ""
}
{
"config":"Motion Tracker",
"operation":"update"
}
{
"running":"Motion Tracker",
"msg":"Background updated."
}
{
"running":"Motion Tracker",
"num":2,
"roi":[
{
"x":0,
"y":0,
"w":0,
"h":0,
"angle":0.0,
"area":0
},
{
"x":0,
"y":0,
"w":0,
"h":0,
"angle":0.0,
"area":0
}
]
}
This function can train and categorize the objects in the green target box in real time, and the trained feature values can be saved on the device for next time use.
Reset
button to clear the existing categories and enter the training mode.+
button to create a new category.Train
button to complete a training session.save&run
button to save the training results on the device and exit the training mode for object recognition and classification.*
{
"function": "Online Classifier",
"args": ""
}
//This instruction will put the device into training mode and extract the features once to store them under the specified class. If class_id does not exist, the class will be created.
{
"config":"Online Classifier",
"operation":"train",
"class_id":1, // Integer (0 ~ N), class ID
"class":"class_1" // String, class
}
{
"running":"Online Classifier",
"msg":"Training [class name] [num of training] times"
}
{
"config":"Online Classifier",
"operation":"saverun",
}
{
"running":"Online Classifier",
"msg":"Save and run."
}
{
"config":"Online Classifier",
"operation":"reset",
}
{
"running":"Online Classifier",
"msg":" Please take a picture."
}
{
"running":"Online Classifier",
"class_num":2, // Number of classes identified
"best_match":"class_1", // Best match class
"best_score":0.83838, // Best Match Score
"class":[ // Scores for each class
{
"name":"class_1",
"score":0.83838
},
{
"name":"class_2",
"score":0.66244
}
]
}
Detect and recognize faces.
{
"function": "Face Recognition",
"args": ""
}
//To create a new face, provide the face_id in order (0 ~ N).
{
"config":"Face Recognition",
"operation":"train",
"face_id":1, // Integer (0 ~ N), 面孔的ID
"name":"tom" // String, Names of faces
}
//For example, there are already 3 faces (0~2), to create a new face, you need to specify the id as 3.
{
"running":" Face Recognition ",
"msg":"Training tom"
}
{
"running":"Face Recognition",
"msg":"Invalid face id"
}
{
"config":" Face Recognition ",
"operation":" stoptrain",
}
{
"running":"Face Recognition",
"msg":"Exit training mode."
}
{
"config":" Face Recognition ",
"operation":"saverun",
}
{
"running":"Face Recognition",
"msg":"Faces saved."
}
{
"config":"Face Recognition",
"operation":"reset",
}
{
"running":"Face Recognition",
"msg":"Reset success"
}
{
"running":"Face Recognition",
"status":"training", // training or missing
"x":0,
"y":0,
"w":0,
"h":0, // Facial recognition bounding box
"prob":0, // Detection confidence rate
"name":0,
}
{
"running":"Face Recognition",
"num":1, // 识别出面部的数目
"face":[
{
"x":0, // 0 ~ 320
"y":0, // 0 ~ 240
"w":30,
"h":30, // Facial recognition bounding box
"prob":0, // Detection confidence rate
"match_prob":0.8, // Matching Confidence Rate
"name": "tom",
"mark":[ // landmarks
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
]
},
]
}
{
"running":"Face Recognition",
"num":1, // Number of faces recognized
"face":[
{
"x":0, // 0 ~ 320
"y":0, // 0 ~ 240
"w":30,
"h":30, // Facial recognition bounding box
"prob":0, // confidence level (math.)
"name": "unidentified",
"mark":[ // landmarks
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
]
},
]
}
Detects the face in the frame and gives a 5-point landmark.
{
"function": "Face Detector",
"args": ""
}
{
"running":"Face Detector",
"num":1, // Number of faces recognized
"face":[
{
"x":0,
"y":0,
"w":30,
"h":30, // Facial recognition bounding box
"prob":0, // confidence level (math.)
"mark":[ // landmark
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
},
{
"x":0,
"y":0
}
]
}
]
}
Detect shapes in a picture and calculate their area. Can recognize square, rectangle, triangle, pentagon, circle. Click the 'Set as background' button to determine the background. The algorithm can adapt to slowly changing backgrounds.
{
"function": "Shape Detector",
"args": ""
}
{
"config":"Shape Detector",
"operation":"update"
}
{
"running":"Shape Detector",
"msg":"Background updated."
}
{
"running":"Shape Detector",
"num":2,
"shape":[
{
"name":"Rectangle", // "unidentified", "triangle", "square", "rectangle", "pentagon", "circle"
"x":0,
"y":0,
"w":0,
"h":0,
"angle":0.0, // Can be used when the shape is square or rectangular
"area":0
},
{
"name":"Rectangle", // "unidentified", "triangle", "square", "rectangle", "pentagon", "circle"
"x":0,
"y":0,
"w":0,
"h":0,
"angle":0.0, // Can be used when the shape is square or rectangular
"area":0
}
]
}
Match any given shape (but the shape should preferably not contain curves), and the uploaded shape will be converted to feature data and saved on the device for next time use.
Set as background
button to determine the background. The algorithm adapts to slowly changing backgrounds。not yet supported
// The shape returned here is the filename of the uploaded template, please note that if the confidence rate is lower than 30%, it will be labeled as unidentified.
{
"running":"Shape Matching",
"num":2,
"shape":[
{
"name":"arrow", // Your customized shape name cannot be recognized when the confidence rate is less than 30
"max_score":83, // Confidence score, if shape is unknown, no
"x":0,
"y":0,
"w":0,
"h":0,
"area":0
},
{
"name":"unidentified", // Your customized shape name is not recognized when the confidence score is less than 30.
"x":0,
"y":0,
"w":0,
"h":0,
"area":0
},
]
}
The audio is captured through the microphone on the device, a real-time FFT (Fast Fourier Transform) is performed and a time-frequency graph is plotted. The green graph below is the RMS of the audio, indicating the current loudness.
None
None
Calling internal models via Python scripts
from json.decoder import JSONDecodeError
import subprocess
import json
import base64
import serial
import time
from datetime import datetime
from PIL import Image
import os
import io
uart_grove = serial.Serial('/dev/ttyS0', 115200, timeout=0.1)
reconizer = subprocess.Popen(['/home/m5stack/payload/bin/object_recognition', '/home/m5stack/payload/uploads/models/nanodet_80class'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
reconizer.stdin.write("_{\"stream\":1}\r\n".encode('utf-8'))
reconizer.stdin.flush()
img = b''
while True:
today = datetime.now()
path = str(today.strftime("%Y_%m_%d") + "/")
newpath = "/media/sdcard/" + path
line = reconizer.stdout.readline().decode('utf-8').strip()
if not line:
break # Process finished or empty line
try:
doc = json.loads(line)
if 'img' in doc:
byte_data = base64.b64decode(doc["img"])
img = bytes(byte_data)
elif 'num' in doc:
for obj in doc['obj']:
uart_grove.write(str(obj['type'] + '\n').encode('utf-8'))
if obj['type'] == "aeroplane":
print('aeroplane ' + today.strftime("%Y_%m_%d_%H_%M_%S"))
if os.path.exists(newpath):
image_path = newpath + today.strftime("%Y_%m_%d_%H_%M_%S") + ".jpg"
img = Image.open(io.BytesIO(byte_data))
img.save(image_path, 'jpeg')
else:
os.mkdir(newpath)
image_path = newpath + today.strftime("%Y_%m_%d_%H_%M_%S") + ".jpg"
img = Image.open(io.BytesIO(byte_data))
img.save(image_path, 'jpeg')
time.sleep(1)
else:
print('Not detect '+ today.strftime("%Y_%m_%d_%H_%M_%S"))
except JSONDecodeError as e:
print("Error: Invalid JSON string")
print("JSONDecodeError:", str(e))