UNSW explores AI infused analytics for dual data lakes

Modern data platform helps UNSW innovate through the pandemic – and plan for the future

Discover just how future-proof the data strategy and Microsoft technologies adopted 12 months ago are proving to be for a small, agile data insights team at UNSW, working remotely during the COVID-19 pandemic.

graphic of an x shape


UNSW staff using Power BI to analyse student enrolments and performance

graphic of two arrows pointing up


To spin up new DevOps pipelines

graphic of two arrows pointing left and right


Power BI dashboards went live for admission in less than three months

graphic of an x shape


Active Power BI dashboards at UNSW

When COVID-19 hit, the Australian university sector was slammed. In a matter of weeks, learning had shifted online, all university staff were working from home, and international students, critical to revenue, were blocked from entering the country.

The University of New South Wales was more fortunate than most: a comprehensive change program introduced 12 months earlier empowered its data insights team to support smarter and faster decision-making across the institution. Fortuitously, the University took a cloud-based approach.

Now it’s using DevOps, twin data lakes and Power BI, while exploring machine learning (ML) and artificial intelligence (AI), to democratise data – and address the challenges of COVID-19.

image of the UNSW campus

Join the team at UNSW as they talk through the journey, process and outcomes of their data innovation and hear what is next.

Watch on demand webinar

Driving change to prepare for an uncertain future

Two years ago, the data insights team at UNSW contemplated the institution’s business-analytics future, with a strong focus on managing student enrolments. As part of the change strategy, the team adopted a suite of Microsoft products to give it powerful new capabilities for managing data and collaborating with other teams in the cloud.

With the rise of COVID-19, those technologies have come to the fore, seamlessly allowing University staff to continuously innovate, even while working from home.

Initially, the change program was motivated by a strong desire to use artificial intelligence and machine learning to unlock more powerful insights, which the University’s legacy data warehouse platform couldn’t support.

Kate Carruthers, Chief Data & Insights Officer, became aware of the evolution in Microsoft’s data offerings after she had lunch with a friend in late 2018.

Microsoft profile picture

Kate Carruthers,

Chief Data & Insights Officer, UNSW

“An old friend of mine [at Microsoft] showed me how Microsoft’s stack has evolved, and upon closer consideration, I realised the company had a really good offering in the data platform space,” Carruthers explains. “I then realised we didn’t just need a data warehouse – we needed a data platform.”

Carruthers knew her small team didn’t have any prior experience with Azure, Power BI and accompanying tools, but she was confident they had the tech know-how and tenacity to quickly adapt. She also connected her team with Microsoft consultants to train them up on the tools and provide support for a smoother rollout. Carruthers’ faith in her team was well-founded.

“I said, ‘Let’s bring it on.’ For me it was like, ‘No guts, no glory,’” recalls Amanda Tjie, Manager, Insights. “We were excited about making changes from the old system to the new world. And we started with small chunks, delivering the top priorities for UNSW to help people make decisions and plan for the future.”

That ‘no guts, no glory’ attitude embraced by the team meant they didn’t want the transformation project to slow down just because they had to work from home.

Microsoft Teams helped the data insights team stay connected to each other and their university stakeholders by providing a single secure hub for meetings, file sharing and project updates.

“Mostly we use Teams for communication and sharing reports,” says Linda Liu, Senior Analyst, Insights. “Currently we push our reports from our Azure data warehouses [Azure Synapse Analytics] to Teams, and we create a channel for everyone who needs it, so they can download the reports they require at any time.”

Introducing new ways to unlock insights from data, despite campus lockdowns

Carruthers’ vision included changing the team from being report writers for the entire university to becoming data engineers who manage the data pipelines and democratise the front end for other users.

“We want the different business units to understand their data and develop their own Power BIs,” she explains. “And do it in a safe, secure environment.”

Carruthers says her team quickly realised the best way to democratise data for decision makers was to make the useful information more accessible. Rather than wading through raw data to find answers, University staff need relevant data curated.

“We’ve got a raw data lake we land the data into, and then we use Databricks to transform it,” she says. “And we put the relevant information into the curated lake, which is what our data scientists are using, not the broad data lake.”

The team is building insights tools with Power BI, which allow people to quickly create visualisations of data to show trends and challenges they need to address.

Carruthers was keen to adopt Power BI because the previous analytics tool was expensive and wasn’t giving end-users what they needed. So, she’s happy the decision was an easy one to make: when the IT department bought an A5 licence from Microsoft, the package included a Power BI professional licence for all staff.

The new Microsoft-powered Power BI reporting services were initially rolled out to people across the entire University. While the newly democratized production model was going live with data analysts from the Faculty of Business, HR and the Deputy Vice-Chancellor Academic teams.

These technologies also help democratise access to information, particularly for those people without deep technical knowledge, notes Liu.

“Normally we provided for the back end. We discussed the basic rules with them, but we didn’t have them help us a lot with the data,” she explains. “Now they help us with dashboards in Power BI because they don’t need many skills: it’s just drag-and-drop. As long as you know the data and the columns, it’s easy to create the dashboard.”

image of the UNSW campus

Join the team at UNSW as they talk through the journey, process and outcomes of their data innovation and hear what is next.

Watch on demand webinar

Specialists from other divisions were also brought in for a few weeks to apply their business rules and explain what the data needed to be used for, notes Alexander Wangsanata, Senior Information Analyst.

“I'm involved in deploying and administrating the Azure platform,” he says. “So, in the beginning, we were trying to find a process that worked for us. We’re not the data owner; it’s coming from other divisions so sometimes we don’t understand what’s been given to us.”

To create excitement around the possibilities of Power BI, Carruthers set up a user group where people could ask questions, and help each other.

“There’s this blossoming of people being able to help themselves and help out others,” she says. “We’ve also developed techniques so they can publish data to Teams. That can drive their Power BIs too, so people have been in control of who can see them. Having that developed and ready to go when COVID-19 happened was really important.”

The various departments have already created about 20 Power BI dashboards since they were introduced to the technology early in the pandemic. Further, they’re doing it safely, because the university’s Microsoft tenancy is encrypted and located in New South Wales, which meets its data jurisdiction and compliance obligations.

Helping people find ways to use the tools effectively is also top of mind for the data team, and so Tjie speaks with users about their challenges and requirements so she can create a prototype.

“We’re currently running training sessions every week,” she says. “We also ask some teams to come in and build with us: we supply the back-end data and then they can build it themselves on Power BI.

“Obviously, different faculties have different needs, so it’s a big advantage that they can actually build their own – it’s all self-service.”

Unlocking better insights by democratising data

Replacing the legacy platform with Microsoft’s Azure, matched with related developer, insights and collaboration tools, was fortuitous for UNSW given the uncertainty around enrolments brought on by the pandemic.

Prior to the data transformation undertaken by the university, decisions about managing enrolments were based simply on making assumptions, says Carruthers.

“The big thing we have brought to the table is fast turnaround of database information for decision-making,” she says. “And it simply would not have been possible with our legacy platform.”

The old platform was so unwieldy it would take three months just to make a minor change, yet in the same timeframe, the data insights team got a whole new system up and running with real-time data.

“Before, they didn’t even know the trends and couldn’t track an application to enrolment, or even look into the numbers day-by-day,” says Tjie. “We managed to give them this capability within three months. Now that they can plan better and make decisions based on our Power BI reports, it opens their eyes to a whole different world.”

The future, says Carruthers, is to develop the next stage of the machine learning and AI strategy, with two proof-of-concepts currently underway.

“The first one is a ‘what-if’ analysis, which uses some machine learning, and the other is to look at contract cheating detection, where ML is used to identify students who cheat by outsourcing their coursework to external parties,” she says. “I suspect the real gems will appear once we see ML in action in our own context. You can have an intellectual understanding of stuff, but you don’t get that visceral understanding of it until you actually do it in your own context and across your own data.”

Additional resources to support your innovation

abstract pattern showing connections of dots

Three ways analytics can help: respond, adapt, and save

closeup of a hand interacting with a surface tablet

Power BI e-book

graphic image of cloud

Free data & analytics cloud training events