Category: Uncategorized

Apple researchers have developed an adapted version of the SlowFast-LLaVA model that beats larger models at long-form video analysis and understanding. Here’s what that means.

The nerdy bits

Very basically, when an LLM is trained to also understand video, it learns to split videos into frames, apply computer vision to extract visual features, analyze how those features change over time, and align all of that with language so it can describe or reason about the video in the form of text.

One very inefficient way to do this is to analyze every single frame of a video, which creates an overwhelming amount of duplicated information, since most frames rarely include significant changes from one to the next.

With this overwhelming amount of duplicated information at hand, it is very easy to blow past the LLM’s context window, which is the maximum amount of information it can retain at once. Once an LLM exceeds its context window, in order for a conversation to keep going, it stops taking older tokens into account to make room for new ones as it predicts each new token.

Of course, there are more efficient ways to train video LLMs (NVIDIA recently published an interesting paper on this), but this is the general idea to keep in mind for Apple’s study.

Apple’s study

As Apple’s researchers explain it in the paper SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding:

“Video large language models (LLMs) integrate video perception into pre-trained LLMs to process videos and generate responses to user commands. Although significant progress has been made, notable limitations remain in existing Video LLMs.”

The limitations, according to them, are threefold:

• Existing models tend to rely heavily on long context windows and huge numbers of frames, which is inefficient and not easily transferable to smaller models;
• Most of them require complex multi-stage training pipelines (often using private datasets) that are hard to reproduce;
• Many are optimized only for video tasks, which limits their usefulness as general-purpose models that also understand images.

To address those limitations, Apple first looked at SlowFast-LLaVA, an open-source model that had already shown promising results by combining spatial and temporal cues through a two-stream setup: a slow stream that looks at fewer frames in higher detail to capture what’s in the scene, and a fast stream that looks at more frames in lower detail to track how things move over time.

First, Apple fine-tuned SlowFast-LLaVA on images, in order to build general visual reasoning capabilities. Then, it was trained jointly on both images and videos (from public datasets), to learn temporal structure without sacrificing image understanding.

The result was SlowFast-LLaVA-1.5 (or SF-LLaVA-1.5), a family of models at 1B, 3B, and 7B parameter scales, that manages to outperform much larger models across a range of video tasks, sometimes “by significant margins,” as noted by the researchers themselves.

In fact, on long-form video benchmarks like LongVideoBench and MLVU, Apple’s model sets new state-of-the-art results across all model sizes, including its smallest, 1B, version.

What’s more, the model also overcomes one of the three shortcomings noted by the researchers, and performs well on image tasks too, including benchmarks for knowledge, math reasoning, OCR, and text-rich scenarios.

The team even tested several video compression strategies, but found that their setup struck the best balance between speed, accuracy, and token count.

Still, there are limitations

With SF-LLaVA-1.5, Apple’s researchers decided that the model would have a maximum input frame length of 128.

This means that whether it is analyzing a clip that is a few minutes or a few hours long, it always maxes out at 128 frames, with 96 evenly spaced frames selected for the fast stream, and 32 evenly spaced frames selected for the slow stream.

With that in mind, the researchers say that:

“This approach may miss some key frames in long-form videos and mislead the model about a video’s playback speed. (…) SF-LLaVA-1.5’s performance can be further improved by tuning all parameters, including the visual encoder. However, we found this is not trivial for Long Video LLMs due to the high GPU memory cost of caching the activation values. Future studies could explore the integration of memory-saving techniques, such as Stochastic BP.”

That said, Apple’s approach rendered it a state-of-the-art model, with the extra chops of being trained exclusively on public datasets. SF-LLaVA-1.5 is now an open-source model available on GitHub and Hugging Face, and you can find the complete study on arXiv.

Below are a few examples of the model in action:

Limited time Apple Watch deals on Amazon

Apple researchers have developed an adapted version of the SlowFast-LLaVA model that beats larger models at long-form video analysis and understanding. Here’s what that means.

The nerdy bits

Very basically, when an LLM is trained to also understand video, it learns to split videos into frames, apply computer vision to extract visual features, analyze how those features change over time, and align all of that with language so it can describe or reason about the video in the form of text.

One very inefficient way to do this is to analyze every single frame of a video, which creates an overwhelming amount of duplicated information, since most frames rarely include significant changes from one to the next.

With this overwhelming amount of duplicated information at hand, it is very easy to blow past the LLM’s context window, which is the maximum amount of information it can retain at once. Once an LLM exceeds its context window, in order for a conversation to keep going, it stops taking older tokens into account to make room for new ones as it predicts each new token.

Of course, there are more efficient ways to train video LLMs (NVIDIA recently published an interesting paper on this), but this is the general idea to keep in mind for Apple’s study.

Apple’s study

As Apple’s researchers explain it in the paper SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding:

“Video large language models (LLMs) integrate video perception into pre-trained LLMs to process videos and generate responses to user commands. Although significant progress has been made, notable limitations remain in existing Video LLMs.”

The limitations, according to them, are threefold:

• Existing models tend to rely heavily on long context windows and huge numbers of frames, which is inefficient and not easily transferable to smaller models;
• Most of them require complex multi-stage training pipelines (often using private datasets) that are hard to reproduce;
• Many are optimized only for video tasks, which limits their usefulness as general-purpose models that also understand images.

To address those limitations, Apple first looked at SlowFast-LLaVA, an open-source model that had already shown promising results by combining spatial and temporal cues through a two-stream setup: a slow stream that looks at fewer frames in higher detail to capture what’s in the scene, and a fast stream that looks at more frames in lower detail to track how things move over time.

First, Apple fine-tuned SlowFast-LLaVA on images, in order to build general visual reasoning capabilities. Then, it was trained jointly on both images and videos (from public datasets), to learn temporal structure without sacrificing image understanding.

The result was SlowFast-LLaVA-1.5 (or SF-LLaVA-1.5), a family of models at 1B, 3B, and 7B parameter scales, that manages to outperform much larger models across a range of video tasks, sometimes “by significant margins,” as noted by the researchers themselves.

In fact, on long-form video benchmarks like LongVideoBench and MLVU, Apple’s model sets new state-of-the-art results across all model sizes, including its smallest, 1B, version.

What’s more, the model also overcomes one of the three shortcomings noted by the researchers, and performs well on image tasks too, including benchmarks for knowledge, math reasoning, OCR, and text-rich scenarios.

The team even tested several video compression strategies, but found that their setup struck the best balance between speed, accuracy, and token count.

Still, there are limitations

With SF-LLaVA-1.5, Apple’s researchers decided that the model would have a maximum input frame length of 128.

This means that whether it is analyzing a clip that is a few minutes or a few hours long, it always maxes out at 128 frames, with 96 evenly spaced frames selected for the fast stream, and 32 evenly spaced frames selected for the slow stream.

With that in mind, the researchers say that:

“This approach may miss some key frames in long-form videos and mislead the model about a video’s playback speed. (…) SF-LLaVA-1.5’s performance can be further improved by tuning all parameters, including the visual encoder. However, we found this is not trivial for Long Video LLMs due to the high GPU memory cost of caching the activation values. Future studies could explore the integration of memory-saving techniques, such as Stochastic BP.”

That said, Apple’s approach rendered it a state-of-the-art model, with the extra chops of being trained exclusively on public datasets. SF-LLaVA-1.5 is now an open-source model available on GitHub and Hugging Face, and you can find the complete study on arXiv.

Below are a few examples of the model in action:

Limited time Apple Watch deals on Amazon

This week Apple hit Apple TV Plus subscribers with a surprise price hike that saw subscription costs leap from $9.99 to $12.99 in the US, disappointing fans of one of the best streaming services around.

The 30% increase is effective immediately for new subscribers, while existing ones have until their next billing cycle. So current subscribers have a minute to decide if they want to continue paying for the service now that it costs roughly $3 more a month.

Authorities in southeastern Spain were forced to temporarily shut down a popular stretch of sand earlier this week after two venomous sea creatures known as “blue dragons” were spotted.

The town of Guardamar del Segura, located just north of Torrevieja on Spain’s Mediterranean Costa Blanca, implemented a swimming ban across all beaches after the rare marine animals — scientifically named Glaucus atlanticus — were discovered in the water near Vivers Beach.

“For motives of citizenry safety, we have banned all swimming on all of the municipal beaches until a new order,” the Guardamar police force said in a Facebook post.

Despite its beautiful appearance, the blue dragon is a potentially deadly sort of sea slug.

Measuring just 3 to 4 centimeters in length, these nudibranchs are rarely seen in the Mediterranean and are more commonly found drifting on the surface of tropical and temperate waters in the Atlantic, Pacific and Indian Oceans.

Their vivid blue coloring and wing-like tendrils give them an ethereal look, but their beauty masks a dangerous defense mechanism.

The blue dragon feeds on venomous prey like the Portuguese man o’ war and concentrates their stinging cells in its own tissues. This makes it more venomous than its prey, capable of delivering painful stings that can cause nausea, vomiting, skin irritation, and acute allergic reactions in humans.

Mayor José Luis Sáez confirmed the sightings and quickly issued a red flag warning, closing beaches while a “preventive device” was deployed to monitor the shoreline for more of the creatures.

Swimmers were told to rinse with saltwater and seek medical attention if they experienced stings or skin irritation.

While rare in this part of the world, their appearance is yet another reminder of the increasing unpredictability of marine ecosystems, possibly linked to climate change and shifting ocean currents.

Guardamar del Segura, known for its sunny climate, rocky cliffs, and golden beaches, draws thousands of tourists each year. The sudden beach closures came at the height of the summer season, but local officials say public safety had to take priority.

Authorities in southeastern Spain were forced to temporarily shut down a popular stretch of sand earlier this week after two venomous sea creatures known as “blue dragons” were spotted.

The town of Guardamar del Segura, located just north of Torrevieja on Spain’s Mediterranean Costa Blanca, implemented a swimming ban across all beaches after the rare marine animals — scientifically named Glaucus atlanticus — were discovered in the water near Vivers Beach.

“For motives of citizenry safety, we have banned all swimming on all of the municipal beaches until a new order,” the Guardamar police force said in a Facebook post.

Despite its beautiful appearance, the blue dragon is a potentially deadly sort of sea slug.

Measuring just 3 to 4 centimeters in length, these nudibranchs are rarely seen in the Mediterranean and are more commonly found drifting on the surface of tropical and temperate waters in the Atlantic, Pacific and Indian Oceans.

Their vivid blue coloring and wing-like tendrils give them an ethereal look, but their beauty masks a dangerous defense mechanism.

The blue dragon feeds on venomous prey like the Portuguese man o’ war and concentrates their stinging cells in its own tissues. This makes it more venomous than its prey, capable of delivering painful stings that can cause nausea, vomiting, skin irritation, and acute allergic reactions in humans.

Mayor José Luis Sáez confirmed the sightings and quickly issued a red flag warning, closing beaches while a “preventive device” was deployed to monitor the shoreline for more of the creatures.

Swimmers were told to rinse with saltwater and seek medical attention if they experienced stings or skin irritation.

While rare in this part of the world, their appearance is yet another reminder of the increasing unpredictability of marine ecosystems, possibly linked to climate change and shifting ocean currents.

Guardamar del Segura, known for its sunny climate, rocky cliffs, and golden beaches, draws thousands of tourists each year. The sudden beach closures came at the height of the summer season, but local officials say public safety had to take priority.

Archaeologists have uncovered a complete plaster cross dating to 1,400 years ago during an excavation in the United Arab Emirates. The Christian symbol finally proves that a series of houses discovered decades ago were part of a monastery.

“This is a very exciting time for us,” Maria Gajewska, an archaeologist at the Department of Culture and Tourism — Abu Dhabi, said in a video. “We never had concrete proof [the houses] were inhabited by Christians.”

Archaeologists have uncovered a complete plaster cross dating to 1,400 years ago during an excavation in the United Arab Emirates. The Christian symbol finally proves that a series of houses discovered decades ago were part of a monastery.

“This is a very exciting time for us,” Maria Gajewska, an archaeologist at the Department of Culture and Tourism — Abu Dhabi, said in a video. “We never had concrete proof [the houses] were inhabited by Christians.”

More than 1,200 people were arrested in a major cybercrime crackdown in Africa, the international police organization Interpol announced on Friday.

Investigators from 18 African countries and the United Kingdom identified some 88,000 victims of fraud and illegal cryptocurrency networks, Interpol said in a statement.

Cybercrime now accounts for more than 30% of all reported crime in West and East Africa, according to the agency.

What else do we know about Interpol’s cybercrime crackdown?

The operation, dubbed “Serengeti 2.0,” took place between June and August. It led to the arrest of 1,209 suspected cybercriminals, recovered $97.4 million (€83.2 million), and took down 11,432 malicious infrastructures.

In one bust, authorities in Angola dismantled 25 cryptocurrency mining centers, where Chinese operators had been mining cryptocurrency.

Angolan officials seized equipment worth an estimated $37 million, “now earmarked by the government to support power distribution in vulnerable areas,” according to Interpol.

In the Ivory Coast, Interpol also said it dismantled a transnational inheritance scam — described as “one of the oldest internet frauds.” Scammers tricked victims into paying fees to claim fake inheritances, causing about $1.6 million in losses.

And in Zambia, investigators targeted a network responsible for an online investment scam that defrauded more than 65,000 victims of an estimated $300 million.

Edited by: Darko Janjevic

More than 1,200 people were arrested in a major cybercrime crackdown in Africa, the international police organization Interpol announced on Friday.

Investigators from 18 African countries and the United Kingdom identified some 88,000 victims of fraud and illegal cryptocurrency networks, Interpol said in a statement.

Cybercrime now accounts for more than 30% of all reported crime in West and East Africa, according to the agency.

What else do we know about Interpol’s cybercrime crackdown?

The operation, dubbed “Serengeti 2.0,” took place between June and August. It led to the arrest of 1,209 suspected cybercriminals, recovered $97.4 million (€83.2 million), and took down 11,432 malicious infrastructures.

In one bust, authorities in Angola dismantled 25 cryptocurrency mining centers, where Chinese operators had been mining cryptocurrency.

Angolan officials seized equipment worth an estimated $37 million, “now earmarked by the government to support power distribution in vulnerable areas,” according to Interpol.

In the Ivory Coast, Interpol also said it dismantled a transnational inheritance scam — described as “one of the oldest internet frauds.” Scammers tricked victims into paying fees to claim fake inheritances, causing about $1.6 million in losses.

And in Zambia, investigators targeted a network responsible for an online investment scam that defrauded more than 65,000 victims of an estimated $300 million.

Edited by: Darko Janjevic