VAR - How can AI and machine learning make football decisions faster?

Has the Video Assistant Referee system (VAR) made football fairer as it was intended to do? Has it helped referees make more correct game-changing decisions by reviewing real-time video evidence? Or has it put more pressure on them and as a result impacted their performance? Has it actually helped improve player behaviour?

There are probably tens of other questions and we will all have different views as there are no black-or-white answers to any of these questions. But it is clear that, like any new technology, VAR is going through its difficult early years as, according to a recent study, 95% of fans said VAR made the experience of watching a game less enjoyable. So, what needs to change for you to one day feel that VAR has actually improved football?

What role can AI and machine learning play?

The main concern about VAR currently is not only the accuracy of the decisions, but also the way it stops the games and the amount of time that it takes to come to a decision. If AI/ML can help resolve these limitations, it will then make VAR do the job it was supposed to do – make football fairer and improve the experience of watching matches.

These limitations of VAR are caused by the fact that it is only used as a video assistance technology for the referees to still make decisions based on real-time judgements. What if the “A” in VAR could become “Automated” instead of “Assistant”? The Video Automated Referee of tomorrow  could be a robot that can make the most accurate decisions within a fraction of a second. That’s what world cups in the 2030s could look like!

The main underlying technology behind the next VAR would lie in video/image recognition, designed to allow computers to obtain, analyse and process information. Specifically, image recognition would be the most important capability for the next VAR, as a video is actually a sequence of images/frames captured and displayed at a given frequency. This will provide all the data that AI-powered machines will need to analyse and make the best decision for every situation. However, this will come with some technical challenges.

Even though various image and facial recognition algorithms have been developed and reached an error rate of just 0.08% in face identification, it will still need to be a technological breakthrough to integrate these with VAR.

Firstly, as the images are collected from different camera angles - even if the algorithm could prove to be efficient and accurate enough to identify and track the player’s gesture in any of the 2D images, it would still be hard to make it accurate enough in the 3D space of the real world. For example, the 3D models of a player’s skeleton need to be built to determine the real-time position of every single part of their body and how this interacts with the ball and other players.

Secondly, the current computer vision algorithms are trained based on deep Learning and networks, which is computationally infeasible and time-consuming as it needs to take all video frames when real-time action develops over hundreds of frames. A common experimental approach is uniformly sampling a small number of video frames and using these to recognise the action. But this method would reduce the accuracy of the results due to the missing information and hence defeat the purpose of improving VAR’s decision-making accuracy. Therefore, how to deal with the trade-off between accuracy and efficiency when producing the identification would be another challenge to improving VAR based on computer vision technology.

The first step?

As an example, the semi-automated offside technology is currently used only as a support tool for the referees to help them make faster, more accurate and more reproducible offside decisions. However, the lead VAR referee still needs to manually validate both the kick point and the offside line after receiving an automated alert.

If the above challenges are overcome, then the offside technology can be fully automated and much faster and, in time, every single situation could be fully automated. With more advanced AI technology involved, e.g., more accurate object detection algorithms and more efficient model training processes, we believe that the future VAR would not require a human referee to monitor the screens. Instead, offsides, fouls and ball positions can be automatically detected in real-time. There will be no constant video replays and discussions anymore - The dream of every football fan nowadays.