How smart displays monitor the host and control the smart home

A smart display is a smart device that consists of a speaker and a screen. It has built-in far-field microphones

actions, dynamics, and for communication withthe user is answered by the voice assistant. These devices also support voice control algorithms: wake word engine, VAD (voice activity detection) and other voice processing algorithms.

The screen shows video, text and graphicinformation, you can use it to make video calls. Some people think of a smart display as an electronic tablet, but they are conceptually different devices. The interface of the smart display is deliberately simplified, since its main purpose is to solve everyday tasks with the help of a voice assistant: control a smart home, planning and managing a calendar, searching for information, setting a timer or an alarm clock.

Smart displays do the same thing as smart speakers, but they add extra functionality to that. For example, here's what the Google Nest Hub Max is capable of:

  • Show videos and movies, but due to the nature of DRM, Netflix appeared on Hub and Hub Max relatively recently, only in July 2020.
  • Display photos.
  • Search information and show search resultson the screen. Interestingly, recipes occupy a large share in the statistics of search queries from smart displays - this gives one of the most popular locations of the device in the house.
  • Make video calls with Duo and Meet, and since August 2020, also Zoom.
  • Translate voice commands into foreign languages ​​in real time, making the translation easier to comprehend when reading from the screen.

If we talk about controlling a smart home, then by selecting the desired device from the list, you can, for example, adjust the brightness of the lamp with one touch. Very comfortably.

An interesting option is also available in the implementationvideo calls, - auto-framing (also known as auto-zoom or auto-pan). With autoframing, the camera focuses on a person and "follows" him while moving (within the camera's field of view, of course). This is achieved using computer vision and image cropping. For the first time on smart displays, autoframing was implemented on the Facebook Portal, and then on the Nest Hub Max. This feature is actively used by Google and Facebook, and users are responding positively to it - this can be seen from the Net Promoter Score.

How it all began

Smart display prototypes without voice assistantsappeared in 2000. Digital electronics maker 3Com (co-founder Robert Metcalfe invented Ethernet) pioneered the development of screen IoT devices. The 3Com Audrey was conceived as a kitchen device and had Internet access, writing and sending emails, and playing audio and video.

In 2006, the first touchscreen device fromChumby for the consumption of media content, which marked the beginning of the line of devices of the same name. Sony Dash, released in 2010, used a platform from Chumby with alarm, radio, Internet access, audio and video playback and a wide variety of applications. Despite the stable operation of the device, in 2017 Sony suspended the work of the servers serving Sony Dash without releasing another analogue.

The main thing that distinguished these devices from current smart displays is the lack of a virtual assistant, gesture control, and voice control.

Sony Dash, 2010

From early adopters to the consumer display boom

Over the past three years, only in the US and Europe has been releasedmore than 30 models of smart displays from more than 12 brands. In June 2017, Amazon launched sales of its first smart display - the Amazon Echo Show, in September of the same year - sales of the Echo Spot, a smart alarm clock with a screen and a voice assistant.

Until the summer of 2018, the online giant wasmonopoly in the smart screen market until Lenovo and JBL released their Google Assistant devices. Google itself announced the Google Home Hub (later renamed the Nest Hub) in October 2018, positioning the device as a hub for smart home control, while its partners using the Google Assistant relied on sound quality.

A month later, Facebook Portal entered the market withAmazon's Alexa voice assistant, and its main user script has been named video calls. Russia also did not lag behind the trend - at the end of 2020, SberDevices announced SberPortal - the first Russian smart display with a system of virtual assistants on board. By the way, the notAnotherOne team was engaged in the design of the Russian portal, and already in April 2021 the device won the most prestigious design awards - Red Dot Design and iF design awards - and with them the recognition of the world design community. Finally, the recently announced update to the iMac line, although it cannot be called a smart display, but a number of functions are gleaned from smart displays - a microphone array of three microphones and six built-in speakers. These innovations allow both to use the device for conferencing and control functions using voice assistant / voice input (starting with the revision of macOS Catalina, users have access to advanced voice control functions, in addition to Siri functionality).

Today we can see the explosive growth of the smart display segment all over the world.

A bit of analytics

Smart display market

First place is occupied by Amazon with 55% market share,followed by Google - 33%, and Facebook - only 2% with its three generations of devices: Facebook Portal, Facebook Portal Plus and Facebook Portal Mini. Smart displays from Lenovo, JBL and LG with built-in Google Assistant account for 6% of the market (no brand breakdown).

The growth in popularity of displays is clearly illustrated by the numbers: in the first quarter of 2018, the share of smart displays was less than 2% of the market, in the first quarter of 2019 - 10%, and by the end of 2020 - already 26%.

* Global Smart Speaker Market Report Q3 2020

As of Q4 2020, smart display sales accounted for just over 25% of total smart speaker sales. In Q1 2018, the share of sales of smart displays is less than 2%, in Q1 2019 - about 10%

Smart display locations

The places where it is customary to place smart displays and speakers is highly predictable: 44% of devices are in living rooms, 37% of devices are in the bedroom, 32% are in the kitchen.

If in living rooms there are more often speakers, since the display function is taken over by the TV, in the kitchen - smart displays, then smart alarms and smart watches dominate in the bedroom.

The image above shows several representatives of thesedevices. They are united by a small basic screen, the resolution of which is sufficient for displaying basic information, as well as a microphone array, speaker and voice assistant. The price tag of such a device can start at $ 50. Amazon initially took the premium segment by adding a camera and quality speakers.

We should also mention the fresh release of smartdisplay from Google - Nest Home Hub 2. Thanks to the built-in mmWave-radar, the device can monitor the phases and duration of the owner's sleep, so it is suitable for placing on the bedside table.

Using an array of microphones, the device is also able to measure breathing rate, snoring duration and number of coughs per night.

Whether the owner's device recognizes by snoring is not yet clear, but we will definitely test this moment.

Displays for the kitchen

In 2018 General Electricannounced a smart cooker hood with a 27-inch touchscreen, two cameras and an on-board Google assistant, the Kitchen Hub. The cooker hood can make video calls, view recipes and play videos on the display. Voice control of the device is carried out through the remote control.

Having already launched Kitchen Hub sales, GE announcedKitchen Hub 2, microwave oven with screen. The developers have placed a third AI camera inside the device, which makes it possible to recognize the ingredients and the readiness of the dishes. Unfortunately, there is no information about the start of sales yet.

Another prototype presented at CES 2019KitchenAid Smart Display - 10-inch KitchenAid Smart Display with Google Assistant from Whirlpool. Its main difference is moisture resistance and an oleophobic coating of the display, thanks to which there are no fingerprints on the surface - a common thing in the kitchen. Alas, the device never went on sale.

Displays for other rooms

Kohler in 2018 announced a smartmirror. Unfortunately, its functionality is only limited to the voice assistant and it is not capable of displaying information. In 2019, several Chinese companies started developing an integrated display smart mirror. Despite the large number of DIY products, mass and commercially successful smart mirrors for the home have not yet happened.

We would like to look into the future and show how we see the concept of a smart mirror that fits perfectly into any bathroom.

Several possible features of this device:

  • Focus on privacy - no cameras + physical button to turn on the microphone or gesture to turn on and off.
  • Electrochromic glass - a variable degree of specularity and transparency depending on the conditions and content that is displayed there.
  • Waterproof, this will help the use of MEMS speakers and piezo microphones.
  • Built-in mmWave radar for recognitionusers, gesture control, recognition of vital activity parameters (temperature, pulse, skin color, pupil condition). The radar can, without the use of cameras, while maintaining the user's privacy, recognize the skeleton and movements of the user, read the heart rate and respiration.

What could happen to the smart display market in the near future?

  • Over time, the display market will become even moresegmented, specialization and use cases for the device will increase: from displays for the kitchen to advanced displays in the bedroom. For example, a smart alarm clock with soft lighting that will gently wake up the user.
  • The emergence of a new segment:there is a shortage of premium devices on the market today. The average cost of a home device is about $ 400. Perhaps there is a need on the part of manufacturers of luxury equipment, including premium audio brands.
  • It is possible to use enabler technologies:displays (microLED, AMOLED, 3D displays, projectors), speakers and microphones (arrays of CMOS speakers, parametric speakers, further increasing the number of microphones in matrices, the use of piezo microphones), radars and sensors (mmWave radars, environmental sensors), connectivity (WiFi 6, 5G).
  • Further growth of the segment with a small amountnew players: only a limited number of companies are ready to partner with Google / Amazon with a large investment volume and low margins. But all the cream today is skimmed by a small number of players with a complete infrastructure of services.

Displays for B2B

Displays in the corporate segment

In many companies, especially those in the Fortune 500,the use of devices with microphones and smart speakers is prohibited, as there is a huge distrust of privacy and data protection. One of the few companies that managed to enter the large corporate segment is Cisco.

Cisco updates its line of smart displays in 2020for conference calls. There are now two models. The youngest model is 23 ″, the older one is 27 ″. The camera is equipped with a physical shutter in both models. The older model supports a stylus, connecting a laptop with parallel charging, AI / dynamic backgrounds, a patented noise reduction system. The cost of the younger one starts from $ 1,490. The cost of the older one starts from $ 5,990.

We should also mention the co-branding initiative from Zoom - Zoom for Home, a line of professional devices for working fromat home. The MSRP of the first device in the line is $ 599. At the same time, the device has a wide range of interfaces, ranging from a 27 ″ touch-screen with pen support, ending with three cameras, an array of eight microphones and stereo speakers. With all this, this is a highly specialized device for video conferencing and presentations - other functionality is cut at the OS level. With no electronics expertise, Zoom Inc. attracted conference solutions manufacturer DTEN as a partner, which paid off - the device was awarded the title of Best of The Best Red Dot design award in 2019.

Hospitality Smart Displays

A big trend is the use of smart speakers and displays in hotels.

Although Amazon launched Alexa for Hospitality back in June 2018, support for this program on the Echo Show is not yet available.

Nest for hospitality - program and initiative fromGoogle aimed at using smart devices in hotel chains. A custom version of Hub Max without a camera is used (in the custom version, the camera is present, in the version for the hotel it is removed) and a special version of the firmware / UI (functionality is limited, administered remotely; the emphasis is on privacy, after the user has checked out, all information about it is erased). The main purpose of the display in this case is to provide the user with information about the services of the hotel and its surroundings, to obtain financial information (invoice for accommodation and additional services).

Another interesting case for hotels wasintroduced by CIRQ + last year. This modular system allows you to control air conditioning, multimedia and lighting systems. The device has a radar to control systems. For example, if the radar is in another room, it adjusts the intensity of the air conditioning in such a way as to save energy costs.

But, as in many other industries, China is ahead of Western competitors - Chinese startup Xiezhu has introduced five smart devices aimed at the hotel segment:

  • Self-check-in display designed to replace the front desk staff.
  • Smart video doorbell for hotels with the ability to let visitors in without getting out of bed.
  • Smart display for rooms, integrated with lighting, climate control and multimedia systems.
  • Smart TV remote control with voice control.

Smart displays in telemedicine

The pandemic has led to increased interest in telemedicine, both in services and in devices for telemedicine examinations.

A number of players (Amwell, H4D, etc.) announced telemedicine kiosks with a smart display at the center. In terms of devices, basic recommendations / instructions are given by a voice assistant, and the process itself takes place under the supervision of a doctor.

Let's allow ourselves to fantasize a little about the topictelemedicine kiosk: a portable system with a smart display as its brain. It is most often required to direct the screen closer to the user, automatic adjustment on the servos is assumed. Array of cameras and sensors to track vital signs / indicators.

  • A portable telemedicine system centered on a smart display.
  • Automatic adjustment of the position of the screen on the servos.
  • An array of cameras for analyzing vital signs, facial expressions.
  • Highly sensitive far-field microphones (analysis of breathing, coughing, wheezing).
  • The platform has enough power to process short-circuit tasks.

Development and popularity of smart display technologiesdue to the unconditional demand for their functionality: from smart home control to sleep monitoring and demonstration of recipes in the kitchen. Total social isolation has further fueled interest in such devices, and the development of the Internet of Things and wireless technologies will make it possible to adapt the use of smart displays for more applied cases, including in difficult conditions of use.

See also:

Smart clothes: what it is and how it works

Scientists have presented a microscope that allows you to see the smallest cellular structures

The animal came to life after 24 thousand years of hibernation in the Siberian permafrost