<\/p>\n\n\n\n
Reflexion.ai is a unique product that combines collaboration, AI-based tagging, and storyboarding in a single web-based Software-as-a-Service (SaaS) application. We help media companies connect everything from task management and review to collaboration, storyboarding, and content metadata extraction and compliance detection using a single easy-to-use tool.<\/p>\n\n\n\n
Our tagging and smart search features help users dig into the video content based on search criteria. Media companies can use these search results to run targeted ads for their advertisers. This capability also helps them create a better sales pitch.<\/p>\n\n\n\n
Reflexion.ai can also help sort pictures and videos by the people who are in it. A critical application is autodetecting compliance components in the content, such as smoking, drinking, and violence.<\/p>\n\n\n\n <\/p>\n\n\n\n We\u2019ve integrated with various external applications to ensure a seamless user experience. Some key integrations are cloud storage, including Dropbox, Blob, and S3 for hassle-free media movement at various stages. We\u2019ve also integrated with editing software such as After Effects, Premiere Pro, Photoshop, and Final Cut Pro to get feedback, manage approvals, and manage files and feedback from anywhere. Reflexion.ai also works with Zapier, further facilitating integration with over 150 Project Management tools, including Jira and Slack.<\/p>\n\n\n\n Probably our most significant partnership is with Azure Machine Learning. Various Azure Machine Learning tools have helped us gain efficiency and scale our operations. We\u2019ll discuss them in depth throughout this article.<\/p>\n\n\n\n Building machine learning models and improving them is an iterative process. This method differs from traditional software development, where the outcome is primarily deterministic. Building and maintaining machine learning models has unique challenges involving standardization, model reproducibility, reusability, and versioning. Azure Machine Learning provides rich, consolidated capabilities for model training, versioning, and deployment.<\/p>\n\n\n\n Azure Machine Learning helps boost our machine learning productivity using AutoML, operationalize at scale with MLOps for multiple model lifecycle management, and provide us with tools to build responsible machine learning solutions. Azure Machine Learning enables us to build an open and flexible platform across development tools, languages, and frameworks, all while maintaining advanced security and governance across the platform.<\/p>\n\n\n\n When we started with Azure Machine Learning, we first turned to their documentation<\/a> for guidance. It reviews Azure Machine Learning and how to set up resources and architecture. It also got us started with Python. We also found it helpful to learn more about Azure Machine Learning\u2019s capabilities<\/a> and review their developer<\/a> resources.<\/p>\n\n\n\n When we started developing Reflexion.ai, we had much to learn. It was our first time building an AI-enabled SaaS application. The journey helped us improve our practices, significantly increasing the efficiency and effectiveness of our work using MLOps.<\/p>\n\n\n\n MLOps is based on the principles and practices of DevOps that increase your workflow efficiency. MLOps combine practices and philosophies for handling applications with machine learning models in a production environment.<\/p>\n\n\n\n Azure Machine Learning provides several MLOps capabilities that enable us to follow the machine learning pipeline efficiently. For example, they help us create reproducible pipelines, develop reusable software environments for training our models, and deploy our models from anywhere\u2014a key component of Reflexion.ai\u2019s collaborative nature.<\/p>\n\n\n\n <\/p>\n\n\n\n As a startup, we had plenty of learning and growing to do in our early days \u2014 especially with machine learning. Let\u2019s examine some events in our journey where we had to adapt and redefine our practices and explore how we leveraged the power of MLOps in Azure Machine Learning to improve the consistency and efficiency of our AI models.<\/p>\n\n\n\n Today, we have more than 20 deep learning audio and vision models in Reflexion.ai. We tune these models based on many datasets for best-in-industry performance. But in the beginning, the team had to work on a limited number of models. So, we had to manually manage the datasets, managing the data in folders then using it for training. Eventually, as we progressed, the number of models increased, and our original process became wildly inefficient. We knew we had to change our methods drastically and introduce automation to keep up.<\/p>\n\n\n\n Critical to any model\u2019s success is the dataset\u2019s quality. Instead of relying on ready-made data collections, we employ a manual process for Reflexion.ai to replicate real-world use cases better. We also have firm guidelines in place to ensure that the collected data is usable.<\/p>\n\n\n\n This data requirement sparked the need for a validation exercise to help weed out abnormalities detected early in the process. This exercise helps us avoid backtracking later when the model fails accuracy tests. Since manually validating every data point is tedious, we explored options to automate the process. We realized Azure Machine Learning could help.<\/p>\n\n\n\n Azure Machine Learning helps us develop scripts to automate initial dataset checks. We also benefit from the collaboration of notebooks and datasets, making it easier for multiple developers to work parallelly. This workflow assures us that the dataset is always in perfect condition, allowing our developers to focus on other aspects of the operation whenever a need for optimization arises.<\/p>\n\n\n\n Data collection is an inherently iterative process, and in turn, the datasets are constantly evolving. Especially for non-textual data, the format of data fed to the model can change the model\u2019s performance. So, our team frequently experiments with different versions of training data to determine the best fit for the model. This experimentation requires a robust procedure for handling different data versions and tracking their performance.<\/p>\n\n\n\n We started doing this experimentation in a rudimentary fashion by creating multiple folders on a single server, which was certainly not going to survive the test of time. Now we\u2019ve entirely ceased doing this and are migrating the entire operation to Azure Machine Learning.<\/p>\n\n\n\n Azure datastores help us dynamically create dataset versions on the fly, attaching proper descriptions that help us use them easily and document the results. It\u2019s reduced our costs, too.<\/p>\n\n\n\n We initially lacked an interface for viewing the datasets quickly whenever needed. Now, with Azure MLFlow, our developers can easily switch between various dataset versions to study the differences. The user interface (UI) removes ambiguity from our operations while helping us compare performance changes.<\/p>\n\n\n\n <\/p>\n\n\n\n During model development, we perform various analyses to evaluate which attributes to focus on. We\u2019d have to perform these analyses for every version of the dataset \u2014 and every iteration of the process. To reduce the manual overhead, we use MLFlow and Azure Databricks to automate the process.<\/p>\n\n\n\n The training process was crucial to improving our efficiency and better handling our machine learning models when we first started. Our first step was to create scoring files and dependencies to adequately document the training and ensure all items in our MLOps pipeline would meet regulatory requirements. When added to Azure Machine Learning\u2019s automated processes, this transparency helped us quickly create accurate models \u2014 and understand how the model was built.<\/p>\n\n\n\n We work with many Python libraries, including PyTorch<\/a>, TensorFlow<\/a>, OpenCV<\/a>, fast,ai<\/a>, PySceneDetect<\/a>, librosa<\/a>, and pydub<\/a>. Working with all these libraries inevitably led to compatibility issues. Sorting these issues out required us to perform manual testing, installing different versions of the libraries with a trial-and-error method. This manual testing is time-consuming and, frankly, it\u2019s a waste of a developer\u2019s time. Azure Resource Manager has helped us save time by automating testing and selecting the best set, which we then create into an environment.<<\/p>\n\n\n\n Azure Machine Learning also helps by creating packages of the libraries and managing what to install on a server. Now, we can quickly create environments using presets. This approach reduces the setup time drastically and lets our developers concentrate on development instead of spending time setting up the environment every time. It also helps us scale up Reflexion.ai without worrying about optimizations.<\/p>\n\n\n\n Three major factors determine the output of any model:<\/p>\n\n\n\n Tuning and finalizing a model requires running multiple experiments on these factors\u2019 various permutations of values and versions.<\/p>\n\n\n\n The best way to experiment and improve a model is by employing version control, as this helps us roll back changes easily in case of a performance drop. The simplest way to do this is using a Git version control system. However, Git is meant mainly for source code versioning. Also, these models are often large, meaning they\u2019re not exactly optimal for Git versioning. Plus, we need to track the other training hyperparameters alongside the code as well.<\/p>\n\n\n\n Integrating with Azure Machine Learning helps us significantly in managing the model versions. We can work with large models and easily adjust the hyperparameters.<\/p>\n\n\n\n Initially, we managed to document our experiments and versioning using elaborate Excel spreadsheets. But as we started scaling up, the challenges increased exponentially. Rerunning the experiment again (and again) by tweaking some factors entailed plenty of setup and documentation. Doing it at scale was challenging \u2014 and it was impossible to do it efficiently. Now, we\u2019re able to perform these reruns seamlessly using Azure ML Studio.<\/p>\n\n\n\n Experiment versioning helps us save the experiment\u2019s iteration during any step of the model building lifecycle. So, rerunning the same experiment configuration and tinkering with the code, parameters, or training set becomes effortless.<\/p>\n\n\n\n Evaluating experiments on the go helps in quickly planning further iterations. The MLFlow interface helps our developers collaborate and track each experiment\u2019s performance. It also enables us to simultaneously compare every version and epoch to determine the best-performing model from the experiments run.<\/p>\n\n\n\n Our team can rerun the same set of model code and training data with various configurations to determine the most optimum parameter values. The collaboration feature helps us track the changes we made, ensuring we don\u2019t rerun the same experiments.<\/p>\n\n\n\n <\/span><\/p>\n\n\n\n MLFlow helps us schedule and queue experiments and gives us options to distinguish various priority levels for each model. This approach has automated our process significantly. So, our developers don\u2019t have to spend time deciding which experiments to run and in which order. The automated process helps our development teams work more efficiently and eliminates hiccups and miscommunications about model prioritization.<\/p>\n\n\n\n When we first started and had a small team and only a few models, it was easy to deploy them whenever they were production-ready. We were also able to modify the models when needed. As the models increased in scale, we chose to automate our testing process using Azure\u2019s continuous integration and continuous deployment (CI\/CD) pipelines.<\/p>\n\n\n\n A CI\/CD pipeline is a deployment pipeline integrated with automation tools for improved workflow. When we conduct them properly, they minimize manual errors and enhance the feedback loops throughout the software development lifecycle, enabling teams to deliver smaller chunks of releases within a shorter time.<\/p>\n\n\n\n Now, we can automatically source artifacts from Azure\u2019s Machine Learning model registry and link it with the pipeline. We can do so while ensuring that updates to production code are subject to appropriate approvals before being updated.<\/p>\n\n\n\n <\/span><\/p>\n\n\n\n When moving from experimentation to production, one of the most challenging tasks is serving deep learning models quickly and easily. Serving deep learning models involves taking a trained model and making it available to support prediction requests. When serving in production, we need to ensure our environment is reproducible, secure, and isolated.<\/p>\n\n\n\n One of the easiest ways to serve deep learning models is by using Nvidia\u2019s Triton Inference Server<\/a>. Triton Inference Server supports all major frameworks, including TensorFlow<\/a>, TensorRT<\/a>, PyTorch<\/a>, ONNX Runtime<\/a>, and even custom framework backends. It can perform real-time inference, batch inference to maximize GPU or CPU use, and streaming inference.<\/p>\n\n\n\n<\/figure>\n\n\n\n
Why We Use Azure Machine Learning<\/h2>\n\n\n\n
Building Reflexion.ai: Implementing MLOps<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
Using Azure Machine Learning and the MLOps Pipeline<\/h2>\n\n\n\n
Preparing and Managing Our Data<\/h3>\n\n\n\n
Data Validation<\/h3>\n\n\n\n
Dataset Versioning<\/h3>\n\n\n\n
<\/figure>\n\n\n\n
Analyzing Data<\/h3>\n\n\n\n
Training Our Models<\/h2>\n\n\n\n
Environment and Resource Management<\/h3>\n\n\n\n
Experiment Management<\/h3>\n\n\n\n
\n
Model Versioning<\/h3>\n\n\n\n
Experiment Documentation and Versioning<\/h3>\n\n\n\n
Experiment Evaluation and Tracking<\/h3>\n\n\n\n
<\/figure>\n\n\n\n
Experiment Scheduling<\/h3>\n\n\n\n
Validating and Deploying Our Models<\/h2>\n\n\n\n
CI\/CD Testing Pipeline<\/h3>\n\n\n\n
<\/figure>\n\n\n\n
Using Triton Inferencing Server<\/h3>\n\n\n\n