The Crown Estate, which is responsible for managing the U.K. seabed, faced the challenge of managing up to 1 petabyte of data originating from various sources. Via a Business-Critical SharePoint solution, ClearPeople connected Microsoft SharePoint with Windows
Azure to share and exchange Big Data with marine renewable energy developers and the public. The processes of managing, releasing, exposing and delivering data have been greatly optimised, with SharePoint acting as the hub orchestrating these processes.
|The Crown Estate
The Crown Estate is a commercial property organisation charged with the managing an £8billion portfolio on behalf of the nation. This includes urban real estate, rural land and almost all of the UK’s seabed.
||The combination of a strong SharePoint Data Management tool and Windows Azure infrastructure allows The Crown Estate to scale up easily to meet demand
| Nigel Spencer,
Head of Information Services
The Crown Estate
Due to its ownership of the seabed, The Crown Estate plays a crucial role in enabling the advancement of the UK offshore renewable energy industry. As well as investing alongside developers, The Crown Estate also utilises its expertise in the field to develop
solutions that drive growth in the industry, whilst also protecting the marine environment. Part of this process has seen The Crown Estate establish a data management repository—the Marine Data Exchange—for managing and storing up to 1 petabyte (one million
gigabytes) of data cultivated from offshore renewable energy developers. Making this data easily accessible provides crucial support for those looking to develop in the marine environment as well as enhancing the sustainability of marine businesses and furthering
our understanding of the country's coastal areas.
The Marine Data Exchange provides anyone with an interest in the marine environment easy access to a wealth of data, ranging from visual and landscape assessments to bird studies. By making this data readily available, the Marine Data Exchange aims to support
the creation of a sustainable blue economy that includes development, conservation and thriving marine businesses.
This data originates from the various renewable energy developers who have a contractual obligation to collect the data. The Crown Estate acts as a custodian of this data, and further has a duty to make such data available to the public; hence the need for
an online resource to publish this critical information.
ClearPeople faced a number of challenges in relation to The Crown Estate’s requirements for the Marine Data Exchange, some of which included:
Data originates from multiple sources and continues to be fed into The Crown Estate.
The authors of the data have no obligation to deliver the data as an online data source; data sets can be delivered via live data connections, Extranet uploads or simple USB drives.
Irrespective of the delivery mechanism, the data does not need to follow the same controls as any other electronic data interchange (EDI) integration, meaning validation against a strict document type definition would be required.
Data needs to be shared with the public, but it was not possible to predict the scale of demand or consumption for this data.
The Crown Estate’s preference was a SharePoint based system. This however raised another set of challenges:
The existing SharePoint infrastructure would not suffice due to the volume of data.
An externally hosted SharePoint infrastructure for public facing demand – given the uncertainty of usage – was seen as too inflexible.
The Crown Estate created a Business-Critical SharePoint (BCSP) solution that uses Microsoft SharePoint 2010 as the hub for large data sets and reports that are produced and managed by over 120 wind farm partners who collect the data and submit it to The Crown
Estate. The BCSP solution connects SharePoint with Windows Azure to share and exchange Big Data - in other words high-volume, and high variety data
1- with energy developers and the public due to its scalability and elasticity.
The Crown Estate have no control over the individual partner’s systems, machines, technologies or data formats - so this could be data coming from ultrasound equipment, seismic equipment, video feeds, via their own data management and line-of-business systems,
as well as quantitative data sets in Excel.
To address this issue, and the estimated 7,500 hours it takes the Crown Estate to manually process this data each year, ClearPeople have developed a common API for integrating and exchanging data with The Crown Estate and its wind farm partners. The common
API requires connections to exchange data in Marine Environmental Data & Information Network (MEDIN) data EDI standard. It is up to the individual partner to either automatically exchange data via this API (“live connections”), or by manually uploading data
to The Crown Estate via a new extranet. In this case, they must supply a correctly formatted XML file in MEDIN format describing the exchanged data. Finally, the partners are in some cases allowed to submit data directly to The Crown Estate via USB drives
(typically a large backlog), using the same common format, in which case The Crown Estate use a “mock on-site data exchange client and validator” to exchange the data, via the common data interface.
In summary, all this data ends up within The Crown Estate domain, and is managed through SharePoint – which in essence surfaces all the metadata and structure fed into it from this large variety of source systems. This is what is known as the Marine Data Exchange
(MDE). Staff at The Crown Estate can therefore use MDE to effectively collaborate, both within internal teams and also external parties, by having all the data relating to the wind farms available through a single SharePoint portal.
The Azure front-end is merely the “icing on the cake” providing the public with access to data, exposed via a search UI. There is also an integration at index and data exchange level between Azure and SharePoint. ClearPeople’s Business-Critical SharePoint solution
exposed data deeply nested within data sets hosted in the various data repositories. By exposing the data and data properties within SharePoint, this could be efficiently accessed by public users through search indexes, facets and refined using a visual geospatial
Bing Maps search.
The Crown Estate saw the clear benefits offered by ClearPeople’s unique solution and SharePoint was connected with Azure infrastructure for:
• Delivery of user interface for the public, integrated with The Crown Estate’s existing web content management system;
• Public web visitors search;
• Geo-spatial indexing of data using Apache Solr;
• Geo-spatial search and integration to Bing Maps; and
• SharePoint Data Management Portal for Marine Data Exchange partners to upload data into the system in consistent formats.
The Data Management Portal in SharePoint provides extranet access for Marine Data Exchange partners to upload and manage their own data – all in consistent formats, saving The Crown Estate valuable time, resources and money by having this entire process automated.
The portal also allows partners and Crown Estate staff to easily locate content via a consistent structure or via SharePoint search features. Data is then automatically exposed within SharePoint, in ‘project areas’ for each renewable energy developer. SharePoint
thereby functions as an effective management hub, while the data continues to reside in external data repositories.
It was extremely important to manage the costs of collecting data. By adopting ClearPeople’s BCSP approach, the project has been effectively implemented, with SharePoint acting as the heart of the operation. The solution also automates the process of ‘releasing
data’ for public consumption once no longer commercially sensitive. The Crown Estate initially had anticipated this would be a manual or semi-manual process. Further, when users request large datasets, the process of delivering this data is automated through
an integrated data delivery mechanism that pushes data from the internal data repositories to a public Azure storage infrastructure acting as a cache until the user has downloaded the data.
Some of the many benefits of implementing a Business-Critical SharePoint solution include the followings:
• The Crown Estate now has an effective solution for managing data in a single repository originating from many parties, through a central and consistent SharePoint user interface.
• Geo-spatial data hidden in XML data structures from renewable energy developer data repositories is now exposed through a user friendly visual Bing Map Search.
• Data which would otherwise sit inside renewable energy developer data repositories is now exposed through a user friendly structure in SharePoint and through filtered searches.
• The processes of managing, releasing, exposing and delivering data have been greatly optimised, with SharePoint acting as the hub orchestrating these processes.
By utilising the new SharePoint Data Management Portal, The Crown Estate will now save a great number of man-hours that were previously spent manually collecting, importing, converting, collaborating, exporting and uploading data to the Marine Data Exchange.
On average, renewable energy partners would send The Crown Estate their data four times per year. Each set of data from each individual renewable energy partner contains a structured index with associated data in deeply nested data folders. Using their traditional
methods of processing this data and previous systems, it is estimated that The Crown Estate would require 1-2 man-days for processing of each data set. With 120 renewable energy partners, this would result in 120 to 240 man-days required four times per year,
or a total of 480 to 960 man-days per year (or more than 2-4 man-years per year just for processing).
By building a collaborative platform, with automated processing of index files, that allows The Crown Estate partners to upload and manage their own data, the requirement for The Crown Estate to manually process datasets has become redundant.
“The combination of a strong SharePoint Data Management tool and Windows Azure infrastructure allows The Crown Estate to scale up easily to meet demand,” says Nigel Spencer, Head of Information Services at The Crown Estate. He continues, “Working with ClearPeople—a
leader in SharePoint based solutions and early adopter of the cloud and Azure— has been a very positive experience.”
Gabriel Karawani, Director and Co-Founder, ClearPeople says: “By delivering the public facing site through Windows Azure’s ‘Platform as a Service’, ClearPeople was able to provide The Crown Estate with a ‘pay as you go’ model which can be scaled up, both in
terms of infrastructure and budget, depending on demand and success. The benefit to the public is a highly accessible system with a user friendly search and Bing Maps interface, giving the user access to very large datasets. It is quite unique.”
The Crown Estate has now adopted SharePoint and Windows Azure as the solution for another similar project, its knowledge network for wave and tidal energy in collaboration with the U.K. Department of Energy and Climate Change.
Spencer says: “Working with Microsoft and ClearPeople—a leader and early adopter of the cloud and Windows Azure— has been a positive experience, so much so that we utilising their support on another major project, our wave and tidal knowledge network.”
Business-Critical SharePoint (BCSP) is a Microsoft partner program that includes top solution and service providers who focus on LOB integration with SharePoint.
1 http://blogs.msdn.com/b/discretemanufacturing/archive/2012/03/10/microsoft-is-big-in-big-data.aspx, March 2012
© 2013 Microsoft Corporation. All rights reserved. This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, [list other trademarks referenced] are either registered
trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. All other trademarks are property of their respective owners.