Yandex Scale 2020: from data marketplace to hybrid clouds

Solution for creating hybrid data clouds

Gazprom Neft and Yandex.Cloud presented

solution for creating hybrid data clouds. With its help, government organizations and private companies will be able to implement Yandex.Cloud technologies, even if they are subject to technical or regulatory restrictions on the use of public data storage and processing services.

  • How will it work?

The user of the IT solution receives fulla ready-to-use hardware and software complex created on the basis of Yandex.Cloud equipment and technologies. The complex is installed in the customer's data center and becomes part of his own platform for working with data. Further service - preventive maintenance, adjustment, updating and other routine maintenance - is carried out according to Yandex.Cloud standards.

  • How will the company's hardware work with the new product?

The new IT product integrates with localcorporate data storage and processing systems and public cloud services. The result is a hybrid cloud that expands the capabilities of the company's digital infrastructure, allows you to set higher security standards and take into account other specifics of a specific user.

During approbation, specialists from GazpromOil ”took part in the development of requirements for the hybrid solution and actually became a partner in the design of its architecture. Therefore, our clients will receive not only a completely ready-to-use hardware and software complex based on Yandex.Cloud technologies, but also a service for supporting and regularly updating the product, taking into account the development of our public platform.

Oleg Koverznev, Chief Operating Officer, Yandex.Cloud

  • What's next planned?

Further development of technology partnershipbetween Gazprom Neft and the Yandex.Cloud platform will allow testing a new digital solution in various industrial operation scenarios. Gazprom Neft is considering the possibility of using a hybrid cloud to develop its computing cluster and developments in the field of artificial intelligence, which are used to search for new oil reserves and remotely control technological operations for its production. Also, additional capacity can be used to improve the efficiency of balancing workloads on the production IT infrastructure.

Every year our factories and oil fields createabout 5 PB of new data. We not only store this information, but make decisions on a daily basis based on its analysis, use it in modeling technological processes and training neural networks. The peculiarity of work in remote regions of Siberia and the Arctic requires large distributed computing power, which would be regulated from one place. The creation of a hybrid cloud solves this problem and provides information security. The product will become a convenient service for industrial companies and other market representatives.

Andrey Belevtsev, Director of Digital Transformation, Gazprom Neft

Four new services for storing and managing data

Service category for storage and managementdata became one of the fastest growing areas of the Yandex.Cloud platform in 2020, surpassing the growth dynamics of services of the traditional leader - virtual machine rental. With the help of cloud data management services, companies, regardless of size and field of activity, solve tasks such as storing and processing data, analyzing and visualizing them.

The services of manageddatabases (Managed Databases) - since the beginning of 2020, the consumption of these services on the Yandex.Cloud platform has tripled, and the number of databases created by users has exceeded 10,000.

  • What happened before?

In our platform Yandex.Cloud was already available by clicking on almost all of the most popular storage and data processing solutions on the market. We've made significant improvements to our data migration scenarios this year. To do this, we added two new services that simplify the transfer of data between any sources - Managed Service for Kafka, and a specialized solution for transferring data between databases - Data Transfer. In the segment of general-purpose databases, the cloud has become even more accessible for Enterprise users with the new Managed Service for SQL Server. Managed Service for Elastic Search, a popular solution that supports full-text search and ad hoc analytics scenarios, has appeared in the family of analytical solutions.

Alexey Bashkeev, head of Yandex.Cloud platform

  • What's coming up now?

Since September 23, users of Yandex.Cloud has opened access to the new Data Transfer service, which allows you to transfer data between DBMSs without stopping the operation of applications, regardless of where they are deployed. Data Transfer helps you quickly and safely migrate databases from other cloud platforms or local databases to Yandex.Cloud managed database services. Also, using Data Transfer, you can move data between different databases on the platform and set up backup.

Also, the Managed service was released.Kafka is a system for streaming data to analytical systems. The service of search and data analysis ElasticSearch and one of the most popular commercial database management systems in the world for work in the ecosystem of Microsoft SQL Server products have been added. In total, 9 managed database services are now available on the Yandex.Cloud platform, covering most scenarios for storing and processing data.

New section of the marketplace with data for business intelligence

In the Yandex.Cloud now has a section called "Geolayers", which contains anonymous geoinformation data from service partners. They can post their content for free or for a fee. Yandex.Cloud clients will access the necessary data in the service interface, analyze it using Yandex DataLens and use it for business purposes. For example, users can assess potential demand for products or prospects for opening outlets, plan supply chain expansions or marketing campaigns.

The section already contains content from two partners -"Center for Spatial Research" on a commercial basis and Rosstat free of charge, as well as two examples of geoanalytics from Yandex - "Audience: Interests and Social Demand" and "Organizations: Supply and Demand". With DataLens, users can combine and analyze all data presets to make business decisions.

We strive to give partners not just technology,and tools for monetizing any intellectual product that can be made or improved using the cloud. Separately, I would like to note that all the content of the "Geolayers" section is exclusively aggregated and anonymized data, for which it is impossible to obtain details for a specific device or user.

Oleg Koverznev, Chief Operating Officer Yandex.Cloud

Who is Yandex.Cloud partner?

Analytics are created in collaboration withRosstat. The cooperation agreement assumes joint work on the preparation of packages of various open data. Vital indicators and other demographic information are already available.

Rosstat publishes largearrays of data available to any user. Thanks to cooperation with Yandex.Cloud, they will gain additional value for researchers, analysts and businesses. The platform's tools allow you to combine sources, make quick analysis, build visual visualizations and, as a result, make quick strategic decisions.

Pavel Smelov, Deputy Head of Rosstat

Center for Spatial Research providedgeoinformation data on the population, households in new buildings, as well as indices of business potential for various areas. In addition, a dashboard for monitoring the network trade of the Russian Federation, which has been running since 2015 and contains an index of the region's development from the point of view of federal retail players, is available free of charge.

According to the general director of the Center for SpatialDenis Strukov's research, analytical tools, indices and expertise, combined with the capabilities of DataLens, is a timely response to the request of the B2B and B2C markets. This, he said, is a cloud version of location intelligence, which has recently been developing abroad, and now in Russia.

Shared access to the Yandex marketplace.Cloud opened in 2019. This is a platform where Cloud customers can directly access business applications from developers and publishers. Today, 47 applications and services are available: from operating systems to information protection tools for genetic analysis.

Yandex.Cloud opens public access to new service for development based on machine learning DataSphere

Yandex.Cloud provides general access to the service for machine learning developers Yandex DataSphere. The service helps companies and individual developers reduce the cost of creating and operating machine learning models, automatically manage the amount and type of computing resources, and reduce the time lost for creating and organizing a development environment. Yandex DataSphere will be openly available from October 1.

Why is this relevant?

Companies' global spending on artificialintelligence is predicted to double over the next four years from $ 50 billion in 2020 to $ 110 billion in 2024. At the end of 2019, Russian companies spent on AI amounted to $ 172 million with a forecast of 30% growth annually. Many Russian companies are already actively using machine learning solutions. For example, in medicine for creating solutions for image analysis, in retail for developing recommendation systems.

Machine learning techniques are becoming more and morepopular business tool around the world. But for many companies it is still not available due to the high entry threshold and the cost of the required computing resources. To solve these problems, we created DataSphere, where you can get a ready-made ML environment at the click of a button. Different types of computing resources are available in it - from classic capacities to GPUs and distributed computing, and billing occurs only for the server power actually consumed while performing your tasks.

Alexey Bashkeev, head Yandex.Cloud platforms

What's new for Yandex.Cloud?

  • Serverless computing technology for developing machine learning models.

Technology automates resource management andallows you to achieve significant savings. In DataSphere, when editing and viewing the code, the computing resources of the CPU or GPU are not involved, the virtual machine of the required type is connected only for the duration of direct calculations (training models, launching, other calculations).

As a result, the user only pays foractually consumed computing resource. The time of editing and viewing the code, the work of an accidentally turned off virtual machine is not charged. DataSphere testing in which 200 users from various industries took part, the downtime of computing power in machine learning development is 50-70%. When using the product, this time will not be charged.

  • Seamless switching between different types of computing resources has been implemented.

This means that within one training scenariomodel, the user can use different types of virtual machines - economical with conventional processors (CPU) and faster with GPU (graphics accelerators). At the same time, the progress of training the model will remain. In most cloud-based machine learning development environments, a learning model can only be calculated on one type of machine.

  • Saving versions of model calculations, including data, code, and states.

This feature makes the machine learning development process more profitable for the business: the achieved learning progress is not lost, it can be reproduced if necessary.

Yandex SpeechKit Pro will help make voice robots smarter and more human

Yandex.Cloud introduced the specialization of the SpeechKit service - Yandex SpeechKit Pro. This is a program for development companies, whose members will have access to new tools for creating robots and voice assistants focused on work in a particular industry or company.

Such robots will be able to recognize words and commandson a specific topic with the highest level of accuracy. The new tools will help to dramatically improve service scenarios in the bank, healthcare and delivery sectors. SpeechKit Pro also allows you to create individual features of a voice robot: intonation and manner of communication.

Why is this relevant?

By 2020, speech synthesis and recognition have becomethe most popular ML-service on the Yandex.Cloud platform. SpeechKit consumption has grown by 120% since the beginning of the year. The number of active projects has exceeded 500. An ecosystem of developers and integrators of solutions has already formed in Russia, who, by order of companies from various fields, create and implement voice robots to help process incoming and outgoing calls, voice control systems in applications and customer service terminals, analysis solutions the effectiveness of business communications.

Today there are more than 20 companies, most ofwhich are regular partners of the Yandex.Cloud platform. According to the partners, over the past two years, the main motives for introducing voice robots in Russian companies have been cost reduction and rapid scaling of solutions.

Together with our partners, we went through a greatway, having made speech technologies from an exotic service an applied business tool in two years. Now we are taking the next step and opening up a new level of Yandex speech technologies for partners. Development companies will have access to the advanced features of SpeechKit, and solution customers will be able to choose the supplier with the most appropriate expertise.

Alexey Bashkeev, head of Yandex.Cloud platform

How will the solution adapt to different business tasks?

Together with the business interest in the possibilities of speechtechnologies, requirements for recognition accuracy in specific scenarios of interaction between voice robots and humans have also grown, the ability to quickly adapt developments for new tasks.

For example, for a shipping companyIt is fundamentally important that the robot does not get confused in evaluating the meaning of the phrases "Transfer the order" or "Enter the order", and for telecommunications companies - that it distinguishes between the phrases "Enable service" and "Disable service" without errors. The priority of a business is accuracy in its area, the ability to develop application experience in a specific business scenario based on objective indicators.

To solve these problems, Yandex.Cloud provides partners with additional development tools as part of the SpeechKit Pro Specialization. Partner companies will now be able to use audio markup, train custom speech recognition models on customer data, monitor speech recognition quality metrics, and adapt recognition models to a specific data stream.

Specialization SpeechKit Pro has already been received by,, Aviation Communication Technologies, Naumen, Robovoice and Voximplant.

Yandex.Cloud platform expanded service ecosystem with serverless computing technologies

Yandex.Cloud has expanded its service ecosystem with proprietary serverless computing technologies. To the four services announced in 2019, two more were added - Yandex API Gateway and Yandex Database in Serverless mode.

What does it mean?

Serverless computing saves companies fromthe costs of solving tasks for the provision and configuration of cloud infrastructure: virtual machines, cloud database servers and applications. These tasks are now performed automatically on the side of the Yandex.Cloud platform.

When using serverless technologiesComputing platform automatically detects, for example, an increase in the number of user requests to the company's application and allocates the resources necessary for stable operation. As soon as the load on the application decreases, the number of capacities involved for its operation decreases. This allows users to switch to a new payment principle for Russian clouds - only on the basis of actual service consumption - and achieve significant savings.

Yandex.Cloud is the first cloud platform in Russia to offer a complete serverless computing ecosystem. We have collected the most necessary data storage and processing technologies for solving urgent business problems and made them available in a serverless mode. Serverless computing is the next phase in cloud computing around the world. This is an opportunity to reduce costs by up to 90%, speed up the time to create and implement new solutions, increase application resilience during periods of peak loads, free companies from the tasks of scaling the service.

Alexey Bashkeev, head of Yandex.Cloud platform

How will it work?

The Yandex.Cloud serverless technology ecosystem now includes six services: Yandex Object Storage - versatile scalable storage solutiondata; API Gateway - a service for creating and managing API; running code as functions - Yandex Cloud Functions; fault-tolerant database management system - Yandex Database; a universal scalable solution for messaging between applications - Yandex Message Queue and Yandex IoT Core Internet of Things service.

For all services of the serverless computing ecosystemYandex.Cloud for the first year (until October 2021) has special rates that allow users to create and host their services for free, not exceeding a certain level of load.

Read also

In the era of ecosystems: how IT giants are turning into interfaces of our everyday life

The Doomsday glacier turned out to be more dangerous than scientists thought. We tell the main thing

GitHub has replaced the term "master" with a neutral equivalent