Creation of a cloud platform for data analysis

GrainCorp contacted Microsoft, who recommended key certified partners from whom, through an evaluation process, GrainCorp selected Insight Enterprise. The Microsoft partner specializes in integration solutions and has 12,000 employees worldwide.

In early 2021, Insight conducted a data asset assessment to understand how GrainCorp handled its analytics and begin building a path to a more efficient, cloud-based operating model.

“I started talking to GrainCorp about machine learning, the different tools their teams use, their pain points, and some pathways I’ve seen working with organizations in the past,” says Julian Wise, Insight Principal Consultant.

“I was thrilled to work with them, especially considering the importance of food security globally. Whether you are looking at the floods in Australia or the war in Ukraine, everything we can do to distribute food more effectively around the world is for the best. “

Based on this assessment, Insight and GrainCorp designed a cloud infrastructure for analytics using a Microsoft Azure Data Lake with Databricks as the primary compute unit.

The infrastructure uses Azure Data Factory to orchestrate integration with data sources in GrainCorp and Azure Key Vault to securely store sensitive information.

“The Insight team was good at guiding us through the whole process,” says James Drummond, chief architect at GrainCorp. “Through this emerged the Databricks platform, which is a leading platform in data science, and a way to increase and decrease to meet computational demand in the short term.

Grain Facilitation Factory
Insight and GrainCorp designed a cloud infrastructure for analytics using Microsoft Azure Data Lake with Databricks (Source: GrainCorp)

“We are no longer paying for more expensive fixed IT infrastructure that needs to be adapted to meet total peak demand. We are able to leverage IT as a utility with greater scalability. “

Optimization of routes and resources

GrainCorp and Insight identified up to 25 use cases that could benefit from the new cloud infrastructure and selected a pilot project: the Transport Optimization Model.

“The [original freight optimisation] the solution worked technically immature, running locally on a single server, but it worked, “says Wise.” There was a lot of manual work and it worked slowly.

“GrainCorp knew there was a way to improve operations, speed up calculations, standardize workflow and integrate it with their systems, to remove manual labor from performing calculations.”

By combining the cloud platform with a proprietary routing algorithm, Insight and GrainCorp quickly prototyped the route optimization model. This included a workflow that updated the central data store every time a data analyst uploaded a spreadsheet with the latest volumes of grain in each silo.

“It only took about 12 weeks to get from the first idea of ​​what the model would look like to a minimum possible product,” says Garner.

“We still try to find the cheapest and most efficient route – we don’t like wasting time and energy,” says Garner.

Leave a Reply

Your email address will not be published. Required fields are marked *