Customer Profile
The European Centre for Medium-Range Weather Forecasts (ECMWF) is an independent intergovernmental organisation with 34 state members. It is both a research institute and a 24/7 operational service. It produces and disseminates numerical weather predictions to its Member States. This data is fully available to the national meteorological services in the Member States. The Centre also offers a catalogue of forecast data for businesses and other commercial customers to purchase worldwide. The supercomputer facility (and associated data archive) at ECMWF is one of the largest of its type in Europe and Member States can use 25% of its capacity for their own purposes.
The organisation was established in 1975 and now employs around 300 staff from more than 30 countries. ECMWF is one of the six members of the Co-ordinated Organisations, which also include the North Atlantic Treaty Organisation (NATO), the Council of Europe (CoE), the European Space Agency (ESA), the Organisation for Economic Co-operation and Development (OECD), and the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT).
Choosing CloudSigma enabled them to provision large volume near real-time weather forecast data sets generated at ECMWF to CloudSigma’s public cloud. There, Member States can access those ECMWF data sets. Without any ECMWF intervention their Member States are able to go directly to the CloudSigma cloud platform, set up their own account, and start receiving the required weather forecast data.
“With CloudSigma, today we can offer to our member states something we could not offer before, which is to have their own virtual machine, with their own software, they take care of their own licences for that deployment, where we co-locate the data that they need.”Ricardo Correa, Network Applications Expert @ ECMWF
The Challenge
The main objective of ECMWF is to send their weather forecast products to the Member States. Since many of those products are of quite large volume (hundreds of gigabytes, even up to a terabyte), sending them over the internet is not an option.
“The internet speed was so slow that by the time the product arrived, the data in the products was useless and was not worth anything. We’re talking about weather forecast data, after all. You either deliver it within one hour or less, or you don’t deliver it at all.”
Therefore, ECMWF was looking for a solution, in which they can process the products and then make them available to the Member States via a virtual machine in the cloud. Since ECMWF does not provide virtual machines themselves, they had to consider a public cloud provider as a partner.
ECMWF searched for contacts via Helix Nebula – the Science Cloud Initiative. After testing with a couple of providers, CloudSigma was the only one able to meet their requirements.
“We did a proof of concept with those providers that agreed to offer a networking test for us, and then the best one was CloudSigma, able to offer the bandwidth we needed.”
The Solution
The key success criteria for the ECMWF chosen cloud provider is to be able to get the data to the site of the Member States on time for it to be useful for any potential third party customers. The reason is that the value of weather forecasts falls very quickly with time.
Hence, the major requirements were fast networking, sustaining up to 3Gbps for real application data transfers from ECMWF with fast storage able to write data quickly. Then, another requirement was fast computing to process that data in time.
“Fast networking and top disk performance are the main reasons for choosing CloudSigma over other providers.”Ricardo Correa, Network Applications Expert @ ECMWF
In addition, the products generated by ECMWF have a very restricted daily schedule. ECMWF is not allowed to release any product before a scheduled time. With CloudSigma, both ECMWF and their customer Member States have their own accounts. ECMWF is able to send the products to the CloudSigma server of the Member State at exactly the scheduled time. After receiving them, the Member State can then post process the products in their virtual machine. Finally, it can expose the results to their own customers.
“We stage these products at CloudSigma, because we get a fantastic network bandwidth to our data center – up to 6 Gbps. As a result, we are able to send the products to our Member States at exactly the scheduled time.”
Using the commercial internet providers, ECMWF could not achieve the performance they needed. The maximum performance was 2 Gbps and there was a lot of packet loss. At CloudSigma the solution was enabled by using a private peering session with GEANT via SWITCH.ch. It had a dedicated bandwidth of 10Gbps (with a commit of 3Gbps 95th percentile).
Another key requirement for the ECMWF use case is to ensure a highly available architecture to avoid single points of failure at the infrastructure level. This was achieved by using the ‘avoid’ functionality for defined servers offered within the CloudSigma functionality. Then, combining this with the node failure tolerant clustered storage system used to drive CloudSigma’s SSD storage offering. The clustered storage system used by CloudSigma offers high availability and an ‘always live’ environment that can survive hardware and software life cycles uninterrupted. This allows the provisioning of true high availability cluster setups.
The Impact
Both ECMWF and their customers are very happy with the service performance. They also use the cloud for other things apart from delivering or receiving this particular weather forecast service. The initial customer that was onboarded today serves as an example for promoting this new ECMWF service. ECMWF is in the process of increasing its collaboration and partnership with CloudSigma in order to benefit further from the expansion of their own service.
“Our customers are very very happy with the service and they are planning to automatically scale horizontally doing all kinds of things.”
A year after the initial cooperation between ECMWF and CloudSigma, ECMWF are planning to benefit from our global exposure. On a closely related project ECMWF has commercial customers in the U.S. requiring data delivery to their sites.
By using our Cloud infrastructure ECMWF is able to transmit the data from within the U.S. lowering network latencies. Therefore, it increases data transfer performance. These customers received service from a proxy host in Canada. Recently, the whole infrastructure was successfully migrated to CloudSigma data center in Washington DC.
“The service is much better than I expected. I really did not expect this networking performance.”Ricardo Correa, Network Applications Expert @ ECMWF
The Future
A next step would be to consider ECMWF customers in Asia Pacific, where they could use CloudSigma’s locations in Manila, Perth or Melbourne and dramatically improve the latency of the current service they’re offering.
“We’re trying to deliver our forecast to the commercial customers worldwide as fast as possible. However, especially for places like Malaysia, China, or Singapore the bandwidth from Europe is really really bad. A small number of dropped packets will bring the performance to very low level. With your new cloud locations in SEA and Australia we will be able to achieve a much higher performance in the whole Pacific area.”
Another planned step is to use a process to automatically create all virtual machines at CloudSigma. Then, use the API for building and provisioning the virtual machines. They will have all the software and tools required by a certain ECMWF customer. In this way customers of ECMWF will be able to choose a customized VM image from the CloudSigma cloud marketplace that has been pre-configured by ECMWF. In addition, the ECMWF tools that help customers interpret and number crunch the obtained data can also be integrated onto the Services Marketplace of CloudSigma.
Download this as a free PDF
We have prepared a free downloadable version of this case study. Feel free to download it for your future reference, and don’t hesitate to contact us with any questions.
Download PDF
- A Cloud for Holography and Cross Reality (CHARITY) - June 18, 2021
- CloudSigma is now a verified Terraform provider - April 30, 2021
- Better performance at lower prices – our U.S. cloud locations - November 21, 2018
- CloudSigma and Apptus Platform partner to offer IT consultancy to APAC clients - October 15, 2018
- Increased pricing for Microsoft licensing - July 16, 2018