RTC League | Be Real-time Every-time

Blogs

Contact us
Contact us

Blog

Me, WebRTC & AI: The Intersection of Innovation and Impact

Greetings to the forward-thinking minds of our generation! This is Mr. WebRTC, a testament to my journey in real-time Audio & Video Communication and Game Streaming. As an experienced WebRTC Engineer and AI Research Scientist, I’ve experienced firsthand how these two technologies  can redefine the possibilities of real-time communication and business operations.

Read More
Setting Up a Professional Grafana Dashboard for LiveKit: A Comprehensive Guide

Creating a customized Grafana dashboard for LiveKit can be a daunting task, especially if you’re starting from scratch. I embarked on this journey to explore and document the process of setting up a fully functional monitoring solution for LiveKit, leveraging Prometheus and Grafana. This guide will walk you through each step, from initial setup to the creation of a professional-grade dashboard.

Read More
The Age of Intelligence: Navigating the Future with AI, AGI, and ASI

As we stand on the brink of a technological revolution, the concept of the Intelligence Age — driven by Artificial Intelligence (AI), Artificial General Intelligence (AGI), and Artificial Super Intelligence (ASI) — promises to redefine human capabilities and societal structures. Inspired by leaders like Sam Altman, my vision is to explore how these advancements can be harnessed to create a prosperous and equitable future.

Read More
Can a Pakistani AI-Startup Become a Billion-Dollar Company?

Starting a company is akin to nurturing a newborn — filled with hope, uncertainty, and relentless effort. As I embark on this journey with RTC League (Real-time Cognitives), I find myself reflecting on the challenges and inspirations that have shaped my path. Inspired by technological visionaries like Steve Jobs, Elon Musk, and Sam Altman, my mission is to demonstrate that even from a country like Pakistan, with its unique set of obstacles, it’s possible to build a billion-dollar tech startup.

Read More
Best Practices for Closing WebRTC PeerConnections:

WebRTC is a fascinating technology that brings real-time communication capabilities to the web. While WebRTC is relatively easy to use, there are many intricacies to it, which, when not understood correctly, can lead to problems. One such issue is closing PeerConnections, especially in complex scenarios like mesh calling. Here, we will delve into the crux of the matter and learn how to overcome this challenge.

Read More
Sam Altman Just Raised $6.6 Billion for OpenAI — But This Is Only the Beginning of the AI Arms Race 🚀

After announcing a record $6.6 billion funding round, OpenAI revealed it also has secured a $4   billion revolving credit facility from major banks. The access to over $10 billion is sorely needed as the artificial intelligence startup burns through cash seeking “to meet growing demand.” As it expands, OpenAI is also undergoing a concentration of power.

Outgoing CTO Mira Murati won’t be replaced, and technical teams will now report directly to CEO Sam Altman himself. With several key executives already gone, “a lot of the opposition to him is gone,” whispers a source from Bloomberg. It seems Sam’s vision is   now uncontested.

Read More
ChatGPT Code Interpret Hacks to Run C++ Code: Navigating the Sandbox

In an era where coding is as much about creativity as  it is about logic, finding new ways to experiment and execute code efficiently has become a pursuit for many developers. My latest adventure with ChatGPT,  a remarkable AI by OpenAI, serves as a testament to this ongoing quest. This post delves into how I  navigated the constraints of a sandbox environment  to run C++ code, revealing insights and tricks that  might just revolutionize your coding sessions.

Read More
Revolutionizing AR/VR Communication: The Role of the Unity WebRTC SDK

Imagine a world where distance is no barrier to immersive, collaborative experiences. Now, stop imagining, because with the Unity WebRTC SDK, that world is becoming a reality. The famous Greek philosopher Epictetus once said, “Progress is not achieved by luck or accident, but by working on yourself daily.” This statement has become my mantra as I endeavored to create the Unity WebRTC SDK for AR/VR communication.

Read More
The Future of Remote Work: Unleashing AI-Powered Communication with WebRTC and Machine Learning

4 years ago, I decided to take a hiatus from Machine Learning, a field that had been a part of my life for an extensive period. The decision was not easy, but it provided me with a different perspective and an opportunity to explore the vast world of technology outside the realm of Machine Learning.

In the interim, I stumbled upon the dynamic technology of WebRTC, which was rapidly transforming the landscape of real-time communication. The more I delved into it, the more I became fascinated by its potential and its pivotal role in powering remote collaboration.

Read More
LiveKit Can Handle Millions of Concurrent Calls — Crazy!

In today’s hyper-connected world, real-time communication is the backbone of our daily interactions. From virtual meetings and online gaming to live streaming events, the demand for seamless, high-quality audio and video communication is skyrocketing. But have you ever wondered what it takes to handle millions of concurrent calls without a hiccup? Enter LiveKit, a real-time audio and video platform that’s redefining scalability with its sophisticated cloud infrastructure.

Read More
Venturing Into the Uncharted: My First IETF Meeting Experience

I am thrilled to announce an exciting new chapter in my professional journey: my first-ever participation in the Internet Engineering Task Force (IETF) meeting from July 22nd to 28th. This global event marks a significant milestone in my journey as a WebRTC Engineer.

Read More
The World’s Fastest Voice Agent with AI, WebRTC, Whisper, and Latency Comparisons

As a software inventor, I’m always exploring new ways to make existing systems faster, more intuitive, and creative. My journey took an exciting turn when I came across a post on Y Combinator’s community by Cerebrium.ai. Their work on reducing latency in voice bots to 500ms caught my attention, and I wondered could I push the limits even further, perhaps even below 300ms?

Read More
OBS Studio and WebRTC Development and Real-time Video Streaming | World’s fastest and real-time streaming solution

Some WebRTC geeks mostly come up with a couple of questions;

  1. How to stream video using OBS Studio over WebRTC?
  2. Is it possible to send Audio Video Packets to WebRTC endpoints  using OBS Studio?

Well, this is possible and let’s have a short demo and explanation of such kind of research work.

 

Read More
Probability and Statistics from Scratch in Julia

Motivation: As part of my personal journey to gain a better understanding of Probability & Statistics, I just follow courses from the world’s best educational Systems such as i.e. MIT, and Stanford. I’ve decided to draw a complete picture of these concepts in Julia Langauge (MIT Labs). Every time, It is basically,  the Internal structure or Inner workings of any “Mathematics or Programming Concept”  that matters a lot for a Data Scientist.

Read More
Probability and Statistics from Scratch in R

Motivation: As part of my personal journey to gain a better understanding of Probability & Statistics, I just follow courses from the world’s best educational Systems such as i.e. MIT, and Stanford. I’ve decided to draw a complete picture of these concepts in R Language. Every time, It is basically, the Internal structure or Inner workings of any “Mathematics or Programming Concept” that matters a lot for a Data Scientist.

Read More
Mastering Coturn Server: From Firewall Evasion to WebRTC Deployment and REST API Development:

In this comprehensive guide, I dive into the world of Coturn Server and its capabilities. From understanding its ability to evade firewalls in “Unpacking the Capabilities of Coturn Server: An Analysis of Firewall Evasion”, to scaling and unlocking the potential of WebRTC in the enterprise with “Scaling & Unlocking the Potential of WebRTC in the Enterprise with CoTURN Server Deployment”, and finally exploring the development and deployment of the CoTURN REST API in “CoTURN REST API Development & Deployment”, this guide covers all you need to know about mastering the Coturn Server.

Read More
Probability and Statistics from scratch in Python:

Motivation: As part of my personal journey to gain a better understanding of Probability & Statistics, I just follow courses from the world’s best educational Systems such as i.e. MIT, and Stanford. I’ve decided to draw a complete picture of these concepts in Python. Every time, It is basically, the Internal structure or Inner workings of any “Mathematics or Programming Concept” that matters a lot for a Data Scientist.

This article contains what I’ve learned, and hopefully, it’ll be useful for you as well!

Read More
Unpacking the Capabilities of Coturn Server: An Analysis of Firewall Evasion

For a comprehensive understanding of CoTURN Server research and development, I invite you to   delve into this guide;

Mastering Coturn Server: From Firewall Evasion to WebRTC Deployment and REST API Development

The widespread use of firewalls to secure networks and protect sensitive information has led to the development of various tools and techniques for bypassing them. One such tool is the Coturn Server, which has gained popularity for its ability to penetrate firewalls. In this article, we will unpack the capabilities of the Coturn Server and examine its efficiency in bypassing firewalls.

Read More

Understanding AI, AGI, and ASI

To navigate the complexities of the Intelligence Age, it’s crucial to understand the distinctions between AI, AGI, and ASI:

  • Artificial Intelligence (AI): Systems designed to perform tasks that typically require human intelligence, such as speech recognition, decision-making, and problem-solving.
  • Artificial General Intelligence (AGI): Machines with the ability to understand, learn, and apply intelligence across a wide range of tasks, much like a human being.
  • Artificial Super Intelligence (ASI): Hypothetical AI that surpasses human intelligence in all aspects, including creativity, wisdom, and problem-solving.

This may turn out to be the most consequential fact about all of history so far. It is possible that we will have superintelligence in a few thousand days (!); it may take longer, but I’m confident we’ll get there.

The Foundations of the Intelligence Age

The journey to the Intelligence Age has been paved by decades of scientific discovery and technological innovation. From melting sand to create computer chips, running energy through them, and developing systems capable of learning and adapting, humanity has built a robust foundation for this new era.

The Role of Deep Learning

At the heart of the Intelligence Age lies deep learning, a subset of AI that focuses on neural networks with many layers. As Sam Altman aptly puts it, “Deep learning worked, got predictably better with scale, and we dedicated increasing resources to it.” This progress has enabled AI systems to handle more complex tasks with greater accuracy and efficiency.

AI models will soon serve as autonomous personal assistants who carry out specific tasks on our behalf like coordinating medical care on your behalf. At some point further down the road, AI systems are going to get so good that they help us make better next-generation systems and make scientific progress across the board.

Embracing Technological Optimism

Despite the challenges, the potential of AI to transform our world is immense. Here’s why a techno-optimistic perspective is essential:

Accelerated Problem-Solving

AI and AGI can address some of the most pressing global issues, from climate change to healthcare disparities. Imagine AI systems that can simulate countless scenarios to find optimal solutions in seconds — something that would take humans decades to achieve.

Democratization of Expertise

ASI has the potential to make expert-level knowledge accessible to all, breaking down barriers related to geography and economics. This democratization can foster innovation from every corner of the globe, empowering individuals to contribute meaningfully to technological advancements.

Enhanced Creativity and Innovation

By automating routine tasks, AI allows humans to focus on creative and strategic endeavors. The synergy between human creativity and AI’s computational prowess can lead to breakthroughs in art, science, and technology that were previously unimaginable.

Economic Growth and Prosperity

Automation and AI-driven productivity can fuel economic growth, creating new industries and opportunities. While concerns about job displacement are valid, history shows that technological revolutions ultimately generate more jobs and enhance living standards.

Improved Quality of Life

AI-driven advancements in medicine, transportation, and daily conveniences can significantly enhance the quality of life. From personalized healthcare to intelligent transportation systems, AI can make our lives safer, healthier, and more efficient.

Navigating Ethical and Social Challenges

While the benefits are vast, the Intelligence Age also brings significant ethical and social challenges. It’s crucial to address these proactively to ensure that AI serves as a force for good.

Ethical Considerations

  • Privacy: Ensuring that AI systems respect and protect individual privacy is paramount.
  • Bias and Fairness: AI algorithms must be designed to minimize biases and promote fairness.
  • Accountability: Establishing clear accountability for AI decisions is essential to prevent misuse and unintended consequences.

Collaborative Governance

Global cooperation is necessary to develop regulations that promote innovation while safeguarding against potential risks. This involves governments, tech companies, and civil society working together to create frameworks that ensure responsible AI development.

Education and Adaptation

As AI reshapes industries, education systems must evolve to prepare individuals for the future workforce. Emphasizing skills like critical thinking, creativity, and adaptability will be crucial in an AI-integrated world.

The Path Forward: Building a Prosperous Future

Drawing inspiration from the relentless pursuit of innovation demonstrated by leaders like Sam Altman, I envision a future where AI, AGI, and ASI are harnessed to create a prosperous and equitable world.

Bridging the Digital Divide

AI can bridge the digital divide by providing access to education, healthcare, and economic opportunities in underserved regions. This democratization of technology can empower individuals and communities, fostering global prosperity.

Fostering Global Collaboration

AI-driven platforms can facilitate unprecedented levels of collaboration across borders. By connecting experts and innovators worldwide, AI can accelerate scientific discoveries and technological advancements.

Encouraging Responsible Innovation

Balancing innovation with responsibility is key to ensuring that AI benefits all of humanity. This involves fostering a culture of ethical AI development, where technological advancements are aligned with societal values and needs.

Conclusion

The Intelligence Age is not just a continuation of technological progress — it represents a fundamental shift in how we harness and interact with intelligence itself. As we embrace AI, AGI, and ASI, we have the opportunity to create a world that is more prosperous, equitable, and innovative than ever before.

However, this journey requires a balanced approach — one that combines technological optimism with ethical responsibility. By learning from the past, honoring personal legacies, and drawing inspiration from visionary leaders, we can navigate the challenges and seize the opportunities of the Intelligence Age.

As someone deeply embedded in the tech community, from founding RTC League in Pakistan to contributing to global advancements in WebRTC and AI , I am committed to leveraging these technologies to bridge gaps and create opportunities for all. Together, we can build a future where technology serves as a bridge, not a barrier — a tool for empowerment rather than division.

Github Facebook-square Reddit Linkedin Twitter

© 2021-2024 RTC League. All rights reserved. 

Terms of Service | Cookie Policy | Privacy Policy

Me, WebRTC & AI: The Intersection of Innovation and Impact

Greetings to the forward-thinking minds of our generation! This is Mr. WebRTC, a testament to my journey in real-time Audio & Video Communication and Game Streaming. As an experienced WebRTC Engineer and AI Research Scientist, I’ve experienced firsthand how these two technologies can redefine the possibilities of real-time communication and business operations.

Setting The Stage

Albert Einstein once said, “The measure of intelligence is the ability to change.” My journey in this dynamic realm started with an innate interest in the adaptability of technology, specifically in WebRTC. Over time, my curiosity was piqued by the possibilities of AI and Machine Learning. I sought to understand how these powerful entities can coalesce to drive innovation in real-time communication.

My foray into this intriguing realm has resulted in commendable achievements, including strategic enhancements to WebRTC platforms and even a U.S. PATENT for ‘METHOD AND SYSTEM FOR TELECONFERENCING USING INDIVIDUAL MOBILE DEVICES.’

In the spirit of constant learning and growth, I encourage everyone to dive deep into the intersection of AI and WebRTC, to question, to innovate, and to create. After all, we’re not just developers or engineers — we’re pioneers on the frontier of technological evolution. And the journey has only just begun.

The Confluence of WebRTC and AI

WebRTC and AI are individually transformative. Yet, their convergence opens up unparalleled avenues for growth and improvement. Below, I delve into four key areas where these technologies can fuse to amplify business outcomes:

1. Enhancing Quality: AI’s adaptive algorithms can dynamically tackle typical communication issues like background noise or poor video quality. By training Machine Learning models for these tasks, we can significantly enhance user experiences during WebRTC communication.

Example: OpenAI’s DALL-E, an AI system, demonstrates the power of generative models that can imagine new concepts. In a similar vein, we can develop AI models to imagine and generate high-quality media streams even under challenging network conditions.

2. Real-Time Analytics: AI offers a unique advantage — the ability to analyze and derive insights from data in real time. Whether it’s converting speech to text, translating languages, or detecting emotions during a call, AI can tremendously enrich WebRTC’s interactive capabilities.

Reference: Google’s ‘Live Transcribe,’ which provides real-time transcription services, is a prime example of harnessing AI’s real-time analytical power.

3. Bolstering Security: AI can work as a smart security agent in WebRTC communication. By learning and detecting unusual patterns, AI can thwart potential security threats, bringing robust security enhancements to real-time communication.

4. Predictive Analytics: AI is a powerful tool for predictive analytics. Coupled with Probability and Statistical Analysis, it can predict future call quality based on current network conditions or even anticipate user behavior.

Quote: As Peter Drucker said, “The best way to predict the future is to create it.” AI empowers us to create a reliable and promising future in the WebRTC landscape.

Pioneering the Future: AI and WebRTC

The synergy between AI and WebRTC signals a future filled with potential and innovation. Recognizing this potential and aligning business strategies to it could provide a significant competitive advantage.

For those new to this intersection, here are a few pointers:

  • Understand the Basics: Start by mastering the fundamentals of WebRTC and AI.
  • Identify the Problem: Identify a challenge where these technologies can offer a solution.
  • Value-First Approach: The focus should be on providing value. Technology is just the means to that end.

As we step into the future, my commitment is to continue to drive the convergence of technology and help organizations unlock the immense potential that WebRTC and AI have to offer. As we all know, the only constant in technology is change, and when we marry evolving technologies like WebRTC and AI, we’re not just keeping up with the changes, but indeed, we’re leading the way.

The Dawn of a New Era: A Conclusion

The journey of merging the realms of AI and WebRTC is much like venturing into uncharted territory. It is filled with challenges and complexities, but also opportunities and rewards. These are the kinds of journeys that change the world, and those who embark on them are the pioneers of technological innovation.

I believe the successful marriage of AI and WebRTC is a game-changer. It has the potential to redefine communication, making it more efficient, secure, and adaptive. It’s not merely an incremental improvement to existing technologies but a paradigm shift that will reshape how we think about real-time communication.

As we look towards the horizon, we see a future where communication is not just real-time but also intelligent, responsive, and secure, thanks to the amalgamation of AI and WebRTC.

As William Gibson famously said, “The future is already here — it’s just not very evenly distributed.” We have the opportunity and the technology to distribute this future more evenly. So let’s embrace this chance, build on this synergy, and create a future that echoes our ambitions and resonates with our vision.

The Expert’s Lens

As an AI and WebRTC veteran, my belief is that this union holds immense potential. The use cases we’ve seen so far are just the tip of the iceberg. The truly transformative impact will surface when we dive deep and explore uncharted territories.

It’s the businesses that understand and harness the power of this synergy that will thrive in the coming years. But remember, technology is just the enabler. The real value lies in how we use it to solve problems, create opportunities, and drive innovation.

In conclusion, the union of AI and WebRTC is more than a new trend; it’s the next step in the evolution of communication. So let’s step into this future, one where we break down barriers, bridge gaps, and bring people closer together, in real-time and intelligently.

From the foundation of my journey — earning the title ‘Mr. WebRTC’ to the development of my seminal U.S. PATENT — to each transformative project I’ve tackled, my experience in this field has underscored one thing: when we unlock the power of technologies like WebRTC and AI, we unlock the potential to create an inclusive, efficient, and secure digital future.

Setting Up a Professional Grafana Dashboard for LiveKit: A Comprehensive Guide

Well, I mostly find something on the internet that is not already cracked. So, I decided to kick start a research work on Grafana Monitoring of LiveKit Server. There was another company (fly.io) that shared its LiveKit monitoring dashboard but somewhat outdated and did not meet the requirements of the latest Livekit states.

Working with LiveKit to create a fully customized Grafana dashboard has been an exciting journey. Over the past three days, I’ve dedicated considerable time and effort to mastering LiveKit’s Stats API, configuring Prometheus, and designing a professional Grafana dashboard. This blog post is meant to share my experience, offering a step-by-step guide that can save time for other developers and researchers. Let’s dive into the process from the very beginning to a polished, professional dashboard.

Introduction

Creating a customized Grafana dashboard for LiveKit can be a daunting task, especially if you’re starting from scratch. I embarked on this journey to explore and document the process of setting up a fully functional monitoring solution for LiveKit, leveraging Prometheus and Grafana. This guide will walk you through each step, from initial setup to the creation of a professional-grade dashboard.

Professional Grafana Dashboard for Livekit
Installation and Development Setup
Setting Up LiveKit on AWS EC2

First, let’s set up LiveKit on an AWS EC2 instance. We’ll use a c5.2xlarge instance to handle heavy operations such as ingress and egress. You can either use LiveKit Cloud (Fully-Managed Server for WebRTC Communication)

Launch an EC2 Instance

  • Choose the Amazon Machine Image (AMI) of Debian.
  • Select the c5.2xlarge instance type.
  • Configure instance details, add storage, and tag the instance as needed.
  • Open the necessary ports in the security group (e.g., 443 for HTTPS, 6789 for Prometheus).

Install Docker and Docker-Compose:

sudo apt update
sudo apt install -y docker.io docker-compose
sudo systemctl start docker
sudo systemctl enable docker

Deploy LiveKit

  • Create a docker-compose.yml file with the following content:
version: '3.7'
services:
  livekit:
    image: livekit/livekit-server
    ports:
      - "7880:7880"
      - "7881:7881"
      - "6789:6789"
    volumes:
      - ./livekit.yaml:/livekit.yaml
    command: /livekit -config /livekit.yaml

Create a livekit.yaml file with the necessary configurations, including the Prometheus endpoint:

prometheus_port: 6789

Start LiveKit using Docker Compose:

docker-compose up -d
Setting Up Prometheus

Next, we’ll set up Prometheus on a separate machine for better fault Isolation.

I have deployed Prometheus on a separate machine to avoid Fault Isolation; It is better when services are on separate machines. If one service fails or requires maintenance, it does not impact the other.

Launch Another EC2 Instance

  • Follow similar steps to create and configure another EC2 instance.

Install Prometheus

sudo apt update
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.29.1/prometheus-2.29.1.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*

sudo cp prometheus /usr/local/bin/
sudo cp promtool /usr/local/bin/
sudo cp -r consoles /etc/prometheus
sudo cp -r console_libraries /etc/prometheus
sudo cp prometheus.yml /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Configure Prometheus

  • Edit the /etc/prometheus/prometheus.yml file for HTTP:
scrape_configs:
  - job_name: 'livekit'
    static_configs:
      - targets: [<'monitor.livekit-dashboard.dev:443'>] # Where you have deployed Livekit Server or endpoint
  • Edit the /etc/prometheus/prometheus.yml file for HTTPS:
scrape_configs:
  - job_name: 'livekit'
    scrape_interval: 10s
    metrics_path: /metrics
    scheme: https
    static_configs:
      - targets: ['IP Address and Port']
    tls_config:
      insecure_skip_verify: true
Setting Up Grafana

Now, let’s set up Grafana, which can be either hosted on Grafana Cloud or self-hosted on an EC2 instance.

Install Grafana

sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Configure Grafana

  • Access Grafana at http://<your-ec2-instance-ip>:3000 and log in with the default credentials (admin/admin).
  • Add Prometheus as a data source:
  • Navigate to Configuration > Data Sources > Add data source > Prometheus.
  • For Grafana Enterprise Cloud, you can either use the search bar to locate “Data Source” or navigate to the “Connections” section.
Grafana Cloud Data Source — Prometheus
  • Enter the Prometheus URL (http://<your-prometheus-instance-ip>:9090) and save.
Prometheus — Data Source
Creating the Grafana Dashboard

With the infrastructure set up, we can now create the Grafana dashboard.

Before starting with the Grafana Dashboard, I found an extensive list of Livekit stats from the Prometheus endpoint. However, I have narrowed it down to the following useful ones:

Livekit Useful Metrics

I created a visual representation using these metrics for Livekit’s live monitoring.

Create a New Dashboard

  • Navigate to Create > Dashboard and add a new panel.
Grafana Dashboard — View
Basic Dashboard

I created a basic dashboard to display simple metrics. This basic LiveKit dashboard provides a comprehensive overview of various metrics related to the LiveKit instance using only Stats manipulation, displayed using Grafana.

Basic Grafana Dashboard — Livekit

Here’s a breakdown of each panel and the information it provides:

Total Rooms

  • Displays the total number of rooms currently active on the LiveKit server.

Total Participants Joined

  • Shows the total number of participants who have joined rooms. The panel currently shows 99 participants, highlighted in red to indicate a potential issue or peak usage.

Total Participants

  • Indicates the total number of participants present in the rooms at the current moment.

Total Publishers

  • Displays the total number of publishers (participants publishing streams) across all rooms. The panel shows 20 publishers.

Total Subscribers

  • Shows the total number of subscribers (participants subscribed to streams). The panel currently indicates 55 subscribers.

Total Published Tracks

  • Represents the total number of tracks (audio or video) published by participants.

Total Subscribed Tracks

  • Indicates the total number of tracks (audio or video) subscribed to by participants.

Quality Score

  • Provides an overall quality score of the media streams, helping to monitor the performance and quality of the LiveKit instance.

CPU Time

  • Displays the CPU usage time for both the LiveKit instance and Prometheus. The panel shows two metrics: 146 seconds for LiveKit and 1532 seconds for Prometheus.

Memory Usage

  • Shows the memory usage for both the LiveKit instance and Prometheus. The panel indicates memory usage in bytes: 108947312 for LiveKit and 82034688 for Prometheus.

Each panel is designed to provide key insights into the performance and usage of the LiveKit server, allowing for effective monitoring and troubleshooting.

Enhanced Dashboard with Time Series

Next, I improved the existing dashboard to have some time-series analysis. So, I enhanced the dashboard by adding time-series queries and configuring alerts.

Enhanced Livekit Dashboard with Time-Series

Add Time Series Queries

{
  "datasource": {
    "type": "prometheus",
    "uid": "ddmi2b881s000e"
  },
  "fieldConfig": {
    "defaults": {
      "custom": {
        "drawStyle": "line",
        "lineInterpolation": "linear",
        "barAlignment": 0,
        "lineWidth": 1,
        "fillOpacity": 0,
        "gradientMode": "none",
        "spanNulls": false,
        "showPoints": "auto",
        "pointSize": 5,
        "stacking": {
          "mode": "none",
          "group": "A"
        },
        "axisPlacement": "auto",
        "axisLabel": "",
        "axisColorMode": "text",
        "axisBorderShow": false,
        "scaleDistribution": {
          "type": "linear"
        },
        "axisCenteredZero": false,
        "hideFrom": {
          "tooltip": false,
          "viz": false,
          "legend": false
        },
        "thresholdsStyle": {
          "mode": "off"
        }
      },
      "color": {
        "mode": "palette-classic"
      },
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "green",
            "value": null
          },
          {
            "color": "red",
            "value": 80
          }
        ]
      },
      "displayName": "Time Series Data",
      "unit": "none"
    },
    "overrides": []
  },
  "gridPos": {
    "h": 9,
    "w": 12,
    "x": 0,
    "y": 0
  },
  "id": 2,
  "options": {
    "tooltip": {
      "mode": "single",
      "sort": "none"
    },
    "legend": {
      "showLegend": true,
      "displayMode": "list",
      "placement": "bottom",
      "calcs": []
    }
  },
  "pluginVersion": "8.0.0",
  "targets": [
    {
      "expr": "your_prometheus_query_here",
      "legendFormat": "Legend Text",
      "refId": "A"
    }
  ],
  "title": "Time Series Example",
  "type": "timeseries"
}

Configure Alerts

  • Set up alerts for high CPU and memory usage. Here is an example of memory usage:
{
  "alertRule": {
    "conditions": [
      {
        "evaluator": {
          "params": [83886080],
          "type": "gt"
        },
        "operator": {
          "type": "and"
        },
        "query": {
          "params": ["A", "5m", "now"]
        },
        "reducer": {
          "params": [],
          "type": "avg"
        },
        "type": "query"
      }
    ],
    "executionErrorState": "alerting",
    "frequency": "1m",
    "handler": 1,
    "name": "Memory Usage Alert",
    "noDataState": "no_data",
    "notifications": []
  }
}
Fully Professional Dashboard

Finally, I created a professional-grade dashboard with detailed titles, custom colors, and fully configured panels.

Fully Customized Professional Dashboard.
  1. Customize Panel Titles and Colors
  • For CPU Time:
{
  "datasource": {
    "type": "prometheus",
    "uid": "ddmi2b881s000e"
  },
  "targets": [
    {
      "expr": "process_cpu_seconds_total{instance='monitor.livekit-dashboard.dev:443', job='livekit'}",
      "legendFormat": "LiveKit CPU Time",
      "refId": "A"
    },
    {
      "expr": "process_cpu_seconds_total{instance='localhost:9090', job='prometheus'}",
      "legendFormat": "Prometheus CPU Time",
      "refId": "B"
    }
  ],
  "title": "CPU Time",
  "type": "timeseries"
}
  • For Memory Usage:
{
  "datasource": {
    "type": "prometheus",
    "uid": "ddmi2b881s000e"
  },
  "targets": [
    {
      "expr": "process_resident_memory_bytes{instance='monitor.livekit-dashboard.dev:443', job='livekit'}",
      "legendFormat": "LiveKit Memory Usage",
      "refId": "A"
    },
    {
      "expr": "process_resident_memory_bytes{instance='localhost:9090', job='prometheus'}",
      "legendFormat": "Prometheus Memory Usage",
      "refId": "B"
    }
  ],
  "title": "Memory Usage",
  "type": "timeseries"
}
  • Save the final dashboard and export the JSON configuration to share with the community.

Fully Customized JSON Script to load:

In conclusion, I am excited to share the results of my extensive three-day research and commitment with you. To ensure the best possible results, I recommend adding the queries for the following metrics from your LiveKit Stats in Prometheus;

  • Total Rooms

livekit_room_total{instance="monitor.livekit-dashboard.dev:443", job="livekit", node_id="ND_k8YVijG2ARvm", node_type="SERVER"}
  • Total Participants
livekit_participant_total{instance="monitor.livekit-dashboard.dev:443", job="livekit", node_id="ND_k8YVijG2ARvm", node_type="SERVER"}
  • Total Participants Joined

livekit_participant_join_total{instance="monitor.livekit-dashboard.dev:443", job="livekit", node_id="ND_k8YVijG2ARvm", node_type="SERVER", state="rtc_connected"}
  • CPU Time
process_cpu_seconds_total{instance="localhost:9090", job="prometheus"}
process_cpu_seconds_total{instance="monitor.livekit-dashboard.dev:443", job="livekit"}
  • Memory Usage
process_virtual_memory_bytes{instance="localhost:9090", job="prometheus"}
process_virtual_memory_bytes{instance="monitor.livekit-dashboard.dev:443", job="livekit"}
  • Total Publishers
livekit_track_publish_counter{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="AUDIO", node_id="ND_k8YVijG2ARvm", node_type="SERVER", state="attempt"}
livekit_track_publish_counter{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="AUDIO", node_id="ND_k8YVijG2ARvm", node_type="SERVER", state="success"}
livekit_track_publish_counter{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="VIDEO", node_id="ND_k8YVijG2ARvm", node_type="SERVER", state="attempt"}
livekit_track_publish_counter{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="VIDEO", node_id="ND_k8YVijG2ARvm", node_type="SERVER", state="success"}
  • Total Subscribers

livekit_track_subscribe_counter{instance="monitor.livekit-dashboard.dev:443", job="livekit", node_id="ND_k8YVijG2ARvm", node_type="SERVER", state="attempt"}
livekit_track_subscribe_counter{instance="monitor.livekit-dashboard.dev:443", job="livekit", node_id="ND_k8YVijG2ARvm", node_type="SERVER", state="success"}
  • Total Published Tracks
livekit_track_published_total{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="AUDIO", node_id="ND_k8YVijG2ARvm", node_type="SERVER"}
livekit_track_published_total{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="VIDEO", node_id="ND_k8YVijG2ARvm", node_type="SERVER"}
  • Total Subscribed Tracks

livekit_track_subscribed_total{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="AUDIO", node_id="ND_k8YVijG2ARvm", node_type="SERVER"}
livekit_track_subscribed_total{instance="monitor.livekit-dashboard.dev:443", job="livekit", kind="VIDEO", node_id="ND_k8YVijG2ARvm", node_type="SERVER"}
  • Quality Score
livekit_quality_score_sum{instance="monitor.livekit-dashboard.dev:443", job="livekit", node_id="ND_k8YVijG2ARvm", node_type="SERVER"}
  • Quality Parameters
LiveKit Quality–Params

Please note that I have modified the queries to prevent potential cybersecurity attacks. You just have to replace `monitor.livekit-dashboard.dev:443` with your livekit stats endpoint in the query section.

I have created a JSON file that you can directly load into your Grafana Dashboard for a fully customized live monitoring experience of your LiveKit server. You can visit this repo for a complete fully functional script.

https://github.com/mail2chromium/LiveKit-Grafana-Dashboard

GitHub Repo

You can also visit my GitHub profile: https://github.com/mail2chromium

Conclusion

Building this Grafana dashboard has been a journey of exploration and dedication. From setting up the infrastructure on AWS to configuring Prometheus and designing a professional Grafana dashboard, every step has been a learning experience. By sharing my configurations and insights, I hope to help others in the community save time and effort. Below are the complete JSON configurations for the dashboards mentioned in this guide. Feel free to use and modify them to suit your needs.

The Age of Intelligence: Navigating the Future with AI, AGI, and ASI

Introduction

As we stand on the brink of a technological revolution, the concept of the Intelligence Age — driven by Artificial Intelligence (AI), Artificial General Intelligence (AGI), and Artificial Super Intelligence (ASI) — promises to redefine human capabilities and societal structures. Inspired by leaders like Sam Altman, my vision is to explore how these advancements can be harnessed to create a prosperous and equitable future.

Many of the jobs we do today would have looked like trifling wastes of time to people a few hundred years ago, but nobody is looking back at the past, wishing they were a lamplighter. If a lamplighter could see the world today, he would think the prosperity all around him was unimaginable. And if we could fast-forward a hundred years from today, the prosperity all around us would feel just as unimaginable. (Sam Altman)

Understanding AI, AGI, and ASI

To navigate the complexities of the Intelligence Age, it’s crucial to understand the distinctions between AI, AGI, and ASI:

  • Artificial Intelligence (AI): Systems designed to perform tasks that typically require human intelligence, such as speech recognition, decision-making, and problem-solving.
  • Artificial General Intelligence (AGI): Machines with the ability to understand, learn, and apply intelligence across a wide range of tasks, much like a human being.
  • Artificial Super Intelligence (ASI): Hypothetical AI that surpasses human intelligence in all aspects, including creativity, wisdom, and problem-solving.

This may turn out to be the most consequential fact about all of history so far. It is possible that we will have superintelligence in a few thousand days (!); it may take longer, but I’m confident we’ll get there.

The Foundations of the Intelligence Age

The journey to the Intelligence Age has been paved by decades of scientific discovery and technological innovation. From melting sand to create computer chips, running energy through them, and developing systems capable of learning and adapting, humanity has built a robust foundation for this new era.

The Role of Deep Learning

At the heart of the Intelligence Age lies deep learning, a subset of AI that focuses on neural networks with many layers. As Sam Altman aptly puts it, “Deep learning worked, got predictably better with scale, and we dedicated increasing resources to it.” This progress has enabled AI systems to handle more complex tasks with greater accuracy and efficiency.

AI models will soon serve as autonomous personal assistants who carry out specific tasks on our behalf like coordinating medical care on your behalf. At some point further down the road, AI systems are going to get so good that they help us make better next-generation systems and make scientific progress across the board.

Embracing Technological Optimism

Despite the challenges, the potential of AI to transform our world is immense. Here’s why a techno-optimistic perspective is essential:

Accelerated Problem-Solving

AI and AGI can address some of the most pressing global issues, from climate change to healthcare disparities. Imagine AI systems that can simulate countless scenarios to find optimal solutions in seconds — something that would take humans decades to achieve.

Democratization of Expertise

ASI has the potential to make expert-level knowledge accessible to all, breaking down barriers related to geography and economics. This democratization can foster innovation from every corner of the globe, empowering individuals to contribute meaningfully to technological advancements.

Enhanced Creativity and Innovation

By automating routine tasks, AI allows humans to focus on creative and strategic endeavors. The synergy between human creativity and AI’s computational prowess can lead to breakthroughs in art, science, and technology that were previously unimaginable.

Economic Growth and Prosperity

Automation and AI-driven productivity can fuel economic growth, creating new industries and opportunities. While concerns about job displacement are valid, history shows that technological revolutions ultimately generate more jobs and enhance living standards.

Improved Quality of Life

AI-driven advancements in medicine, transportation, and daily conveniences can significantly enhance the quality of life. From personalized healthcare to intelligent transportation systems, AI can make our lives safer, healthier, and more efficient.

Navigating Ethical and Social Challenges

While the benefits are vast, the Intelligence Age also brings significant ethical and social challenges. It’s crucial to address these proactively to ensure that AI serves as a force for good.

Ethical Considerations
  • Privacy: Ensuring that AI systems respect and protect individual privacy is paramount.
  • Bias and Fairness: AI algorithms must be designed to minimize biases and promote fairness.
  • Accountability: Establishing clear accountability for AI decisions is essential to prevent misuse and unintended consequences.
Collaborative Governance

Global cooperation is necessary to develop regulations that promote innovation while safeguarding against potential risks. This involves governments, tech companies, and civil society working together to create frameworks that ensure responsible AI development.

Education and Adaptation

As AI reshapes industries, education systems must evolve to prepare individuals for the future workforce. Emphasizing skills like critical thinking, creativity, and adaptability will be crucial in an AI-integrated world.

The Path Forward: Building a Prosperous Future

Drawing inspiration from the relentless pursuit of innovation demonstrated by leaders like Sam Altman, I envision a future where AI, AGI, and ASI are harnessed to create a prosperous and equitable world.

Bridging the Digital Divide

AI can bridge the digital divide by providing access to education, healthcare, and economic opportunities in underserved regions. This democratization of technology can empower individuals and communities, fostering global prosperity.

Fostering Global Collaboration

AI-driven platforms can facilitate unprecedented levels of collaboration across borders. By connecting experts and innovators worldwide, AI can accelerate scientific discoveries and technological advancements.

Encouraging Responsible Innovation

Balancing innovation with responsibility is key to ensuring that AI benefits all of humanity. This involves fostering a culture of ethical AI development, where technological advancements are aligned with societal values and needs.

Conclusion

The Intelligence Age is not just a continuation of technological progress — it represents a fundamental shift in how we harness and interact with intelligence itself. As we embrace AI, AGI, and ASI, we have the opportunity to create a world that is more prosperous, equitable, and innovative than ever before.

However, this journey requires a balanced approach — one that combines technological optimism with ethical responsibility. By learning from the past, honoring personal legacies, and drawing inspiration from visionary leaders, we can navigate the challenges and seize the opportunities of the Intelligence Age.

As someone deeply embedded in the tech community, from founding RTC League in Pakistan to contributing to global advancements in WebRTC and AI , I am committed to leveraging these technologies to bridge gaps and create opportunities for all. Together, we can build a future where technology serves as a bridge, not a barrier — a tool for empowerment rather than division.

Can a Pakistani AI-Startup Become a Billion-Dollar Company?

Introduction

Starting a company is akin to nurturing a newborn — filled with hope, uncertainty, and relentless effort. As I embark on this journey with RTC League (Real-time Cognitives), I find myself reflecting on the challenges and inspirations that have shaped my path. Inspired by technological visionaries like Steve Jobs, Elon Musk, and Sam Altman, my mission is to demonstrate that even from a country like Pakistan, with its unique set of obstacles, it’s possible to build a billion-dollar tech startup.

Cognitive AI is a type of artificial intelligence (AI) that mimics human thought processes and learning abilities

Drawing Inspiration from the Greats

From a young age, I was captivated by the stories of innovators who changed the world. Steve Jobs, whose biography by Walter Isaacson remains a cornerstone of my inspiration, taught me the importance of vision and relentless pursuit of excellence. His ability to blend technology with design and user experience showed me that innovation isn’t just about creating something new but making it meaningful and accessible.

Elon Musk, detailed in Ashlee Vance’s “Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future,” embodies the spirit of audacity and resilience. Musk’s ventures into space exploration and electric vehicles illustrate how thinking big and challenging the status quo can lead to groundbreaking advancements. Sam Altman’s insights into AI and the future of technology have further fueled my passion for innovation, emphasizing the potential of artificial intelligence to solve complex global issues.

I immerse myself in their stories — reading their biographies, watching their interviews, and learning from their experiences. These mentors, though not physically present, have significantly influenced my approach to entrepreneurship and technology.

Honoring My Mother’s Legacy

Behind every entrepreneur lies a foundation of personal experiences and lessons. For me, that foundation is deeply rooted in the resilience and determination of my late Mother. She was an MBBS doctor who chose to leave her esteemed career to build a nurturing environment for our family. Facing immense opposition from her own family, she demonstrated unwavering patience and strength. Her decision to prioritize our family’s culture and well-being over her professional aspirations taught me that true leadership is about sacrifice and perseverance.

I remember vividly the day I asked my beloved mother why she left her MBBS job. With pure confidence, she replied, “I feel that if I continue my job, I may not be able to build the culture of my family and kids.”

And by God, my mom then sacrificed her moments for us, and for us only!!!

When our beloved mother passed away just as my name began to gain recognition in the tech world, her legacy remained a guiding force. She was more than just a parent; she was a beacon of dedication and love. Her influence continues to inspire me to push forward, no matter the challenges.

The Birth of RTC League

In January 2019, I joined forces with Shahmeer Amir, Pakistan’s third-ranked ethical hacker, to started working as an employee on a startup venture. For over five years, Shahmeer was not just a mentor but like an elder brother to me. Together, we navigated the tumultuous waters of research work in that startup. However, in April 2024, I made the difficult decision to part ways with Shahmeer, despite the lack of support during that transition. This setback, though disheartening, reinforced my resolve to forge my own path. I wish him best of luck.

Determined to create something impactful, I partnered with my elder brother, Muhammad Aatif Bashir Choudhary, a seasoned telecom professional with over a decade of experience in customer care and management at Etisalat Dubai. He is just not my elder brother but a great friend and is a hero of mine.

The name RTC League holds sentimental value — it was a name our late mother often mentioned, and we decided to honor her memory by keeping it.

Building a Team from Scratch

One of the core beliefs that drive RTC League is that with the right leadership, any person can be trained to achieve greatness. Convincing my brother to join this venture was no small feat. His extensive experience in telecommunications complemented my technical expertise in WebRTC and AI full-stack development. Together, we envisioned a company that could innovate and lead in real-time communication technologies, with real-time AI cognitives.

Understanding the importance of fresh perspectives and the potential of young minds, we decided to hire recent graduates and current students. We brought on a team of seven fresh graduates and two BSCS students in their sixth semester. While we faced setbacks — losing one student during the transition — it was a valuable lesson in mentorship and team dynamics. Despite these challenges, our small yet dedicated team managed to develop groundbreaking products backed by the Punjab Information Technology Board (PITB) in Pakistan.

Valuing teamwork above all, we focused on building a great team culture around these fresh graduates. By providing them with guidance, training, and a supportive environment, we set an example of how a team of newcomers can achieve remarkable feats. It reinforced my belief that you only need good leaders to build a great research lab around fresh graduates.

Achievements Amidst Adversity

In just six months, our team developed two remarkable products:

  1. Multilingual Audio-Video Calling Platform: A real-time, web-based platform enabling seamless communication across different languages. This platform leverages AI to mimic human voice in any desired language in real-time.
  2. Walkthrough Desktop Controller: A solution allowing users to control their entire laptop using a mobile phone using real-time voice and AI.
  3. We have also built an extensive research based product: RTC Studio — OBS Studio and WebRTC to provide lowest latency using Ant Media Server. It is available on Antmedia Market Place.

Facing challenges in sales, we pivoted two months ago to develop a SaaS-based leads generation platform. This new product leverages WebRTC conversational agents for inbound and outbound calling using Twilio, Vonage Technology, and Telnyx. Already, it has garnered admiration from several potential investors, signaling promising growth prospects.

Overcoming Financial Hurdles

Like many startups, we faced financial constraints. During the early stages, I had to borrow funds from friends to pay our team, ensuring that the company could continue to operate despite the challenges. This experience taught me the importance of knowing when to ask for help and being willing to accept it when offered. These financial hurdles only strengthened my commitment to RTC League. I have the grit spirit, just like Elon Musk; no matter what we are facing, we have to stand tall and build amazing things. I am prepared for the worst but remain optimistic about our future.

Lessons Learned

Throughout this journey, I’ve learned invaluable lessons that have shaped not only our company but also my personal growth:

  1. Resilience: Facing setbacks head-on and using them as learning opportunities is crucial. When I faced opposition from mentors and the challenges of parting ways with a close collaborator, standing tall and accepting the changes was essential.
  2. Teamwork Above All: Valuing teamwork and building a strong team culture around fresh graduates has been instrumental in our success. Combining strengths with my brother and leveraging the fresh perspectives of our team has proven that with good leadership, any person can achieve great things.
  3. Innovation and Adaptability: Continuously adapting and pivoting to meet market needs is vital. Our shift to developing a SaaS-based leads generation platform is a testament to our ability to innovate and respond to challenges.
  4. The Importance of Family and Friends: The love and support of family and friends are paramount. My brother’s relentless efforts and the inspiration from my late mother have been the backbone of our journey.
  5. Knowing When to Ask for Help: Learning when to seek assistance and accepting help when offered can make the difference between success and failure. Borrowing funds from friends to support the team was a humbling experience that underscored the importance of community.
  6. Building a Great Team Culture: Hiring fresh graduates and setting an example by properly building a great team culture around them has shown that talent can be nurtured. With the right environment and guidance, new entrants can make significant contributions.
  7. Grit and Determination: Embracing the grit spirit, much like Elon Musk, means persisting in the face of adversity. No matter what challenges we face, we stand tall and continue to build amazing things.

A Vision for the Future

I firmly believe that RTC League will become a billion-dollar company using WebRTC and AI-based products by 2029, all while operating from Pakistan. This vision isn’t just about personal success; it’s about setting an example. I want to prove that even in a country with limited resources, inconsistent internet access, and restricted platforms like Twitter, it’s possible to build a great startup.

Our journey so far has been a testament to what can be achieved with determination, a strong team, and a clear vision. My brother’s relentless efforts in conveying our hard work and technological advancements to business leaders across Pakistan have been instrumental in our progress.

Conclusion

Starting RTC League has been a journey of faith, resilience, and relentless hard work. While the road is steep and fraught with challenges, I am driven by a vision to not only succeed but to inspire others in similar circumstances. Our story is proof that with determination, a supportive team, and a willingness to defy the odds, it’s possible to achieve greatness.

In honoring my mother’s legacy and drawing inspiration from the greats like Steve Jobs and Elon Musk, I am committed to building a future where technology bridges gaps and creates opportunities for all. I invite you to join us on this journey — to witness how a startup from Pakistan is poised to make a global impact. Together, we can break barriers, challenge norms, and pave the way for future innovators from all corners of the world.

Stay tuned for more updates!

Best Practices for Closing WebRTC PeerConnections:

A Comprehensive Guide for iOS and Android

WebRTC is a fascinating technology that brings real-time communication capabilities to the web. While WebRTC is relatively easy to use, there are many intricacies to it, which, when not understood correctly, can lead to problems. One such issue is closing PeerConnections, especially in complex scenarios like mesh calling. Here, we will delve into the crux of the matter and learn how to overcome this challenge.

The Challenges Faced

I once spent more than a week trying to debug a seemingly straightforward issue. In a mesh calling scenario involving 8 participants from different platforms — Android and iOS, my application would become unresponsive and eventually crash. This was specifically happening on the iOS side when I was trying to close a PeerConnection while the other participants were still in the ‘connecting’ state. What seemed like a resource consumption crash was actually an issue with the main UI thread being blocked due to WebRTC background tasks.

Crash Catalog:

Over the past six months of intense and passionate development, we hadn’t experienced a crash quite like this one. It was an anomaly that had us puzzled. What was even more perplexing was that the crash reports indicated an issue of resource consumption — a possibility that hadn’t occurred to us in the context of a seemingly innocuous task of closing peer connections.

The iOS watchdog, meant to prevent any app from hogging system resources, ended up terminating our application. This signaled to us that something under the hood was indeed amiss.

Here are some screenshots of the crashes:

Test Flight Crash Report:


Date/Time:           2023-06-21 15:53:37.6520 +0500
Launch Time:         2023-06-21 15:43:18.2579 +0500
OS Version:          iPhone OS 16.5 (20F66)
Release Type:        User
Baseband Version:    4.02.01
Report Version:      104

Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Termination Reason: FRONTBOARD 2343432205
<RBSTerminateContext| domain:10 code:0x8BADF00D explanation:scene-update watchdog transgression: application<com.younite.development>:2048 exhausted real (wall clock)
time allowance of 10.00 seconds
ProcessVisibility: Foreground
ProcessState: Running
WatchdogEvent: scene-update
WatchdogVisibility: Foreground
WatchdogCPUStatistics: (
“Elapsed total CPU time (seconds): 3.380 (user 3.380, system 0.000), 5% CPU”,
“Elapsed application CPU time (seconds): 0.049, 0% CPU”
) reportType:CrashLog maxTerminationResistance:Interactive>

Triggered by Thread: 0

Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0 libsystem_kernel.dylib 0x20bb88558 __psynch_cvwait + 8
1 libsystem_pthread.dylib 0x22c9d0078 _pthread_cond_wait + 1232
2 WebRTC 0x109f92a20 0x109e4c000 + 1337888
3 WebRTC 0x109f92900 0x109e4c000 + 1337600
4 WebRTC 0x109ebe20c 0x109e4c000 + 467468
5 WebRTC 0x109ebe10c 0x109e4c000 + 467212
6 WebRTC 0x109ebda38 0x109e4c000 + 465464
7 WebRTC 0x109ebda14 0x109e4c000 + 465428
8 WebRTC 0x109f6b52c 0x109e4c000 + 1176876
9 libobjc.A.dylib 0x1c5cb60a4 object_cxxDestructFromClass(objc_object*, objc_class*) + 116
10 libobjc.A.dylib 0x1c5cbae00 objc_destructInstance + 80
11 libobjc.A.dylib 0x1c5cc44fc _objc_rootDealloc + 80
12 libobjc.A.dylib 0x1c5cb60a4 object_cxxDestructFromClass(objc_object*, objc_class*) + 116
13 libobjc.A.dylib 0x1c5cbae00 objc_destructInstance + 80
14 libobjc.A.dylib 0x1c5cc44fc _objc_rootDealloc + 80
15 WebRTC 0x109f7302c 0x109e4c000 + 1208364
19 libswiftCore.dylib 0x1c6d11628 _swift_release_dealloc + 56
20 libswiftCore.dylib 0x1c6d1244c bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1>>::doDecrementSlow<(swift::PerformDeinit)1>(swift::RefCountBitsT<(swift::RefCountInlinedness)1>, unsigned int) + 132
21 libswiftCore.dylib 0x1c6d012c8 swift_arrayDestroy + 124
22 libswiftCore.dylib 0x1c6a1c2b0 _DictionaryStorage.deinit + 468
23 libswiftCore.dylib 0x1c6a1c31c _DictionaryStorage.__deallocating_deinit + 16
24 libswiftCore.dylib 0x1c6d11628 _swift_release_dealloc + 56
25 libswiftCore.dylib 0x1c6d1244c bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1>>::doDecrementSlow<(swift::PerformDeinit)1>(swift::RefCountBitsT<(swift::RefCountInlinedness)1>, unsigned int) + 132
27 libswiftCore.dylib 0x1c6d11628 _swift_release_dealloc + 56
28 libswiftCore.dylib 0x1c6d1244c bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1>>::doDecrementSlow<(swift::PerformDeinit)1>(swift::RefCountBitsT<(swift::RefCountInlinedness)1>, unsigned int) + 132
30 libsystem_blocks.dylib 0x22c9c5134 _call_dispose_helpers_excp + 48
31 libsystem_blocks.dylib 0x22c9c4d64 _Block_release + 252
32 libdispatch.dylib 0x1d40eaeac _dispatch_client_callout + 20
33 libdispatch.dylib 0x1d40f96a4 _dispatch_main_queue_drain + 928
34 libdispatch.dylib 0x1d40f92f4 _dispatch_main_queue_callback_4CF + 44
35 CoreFoundation 0x1cccb3c28 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16
36 CoreFoundation 0x1ccc95560 __CFRunLoopRun + 1992
37 CoreFoundation 0x1ccc9a3ec CFRunLoopRunSpecific + 612
38 GraphicsServices 0x20815f35c GSEventRunModal + 164
39 UIKitCore 0x1cf0276e8 -[UIApplication _run] + 888
40 UIKitCore 0x1cf02734c UIApplicationMain + 340
41 Younite 0x101509d00 main + 64
42 dyld 0x1ec19adec start + 2220

While Debugging:

These crash reports provide a snapshot of the challenge we were up against. Through persistence and deep-dive analysis, we uncovered the core of the issue — a hiccup in our understanding and usage of the WebRTC protocol.

The Importance of Connection State

Before discussing the strategies to close peer connections, it’s important to understand the different states a connection can have. In WebRTC, the RTCPeerConnection.connectionState property can tell you about the current connection state. The possible states are:

  • new: The connection has just been created and has not completed negotiation yet.
  • connecting: The connection is in the process of negotiation.
  • connected: The connection has been successfully negotiated and active data channels are open.
  • disconnected: One or more transports are disconnected.
  • failed: One or more transports has terminated or failed.
  • closed: The connection is closed.

Before calling RTCPeerConnection.close(), it’s crucial to ensure that the connection is either connected or failed. Closing connections in transitional states (connecting or disconnected) can lead to issues and can cause the application to become unresponsive.

Unraveling PeerConnection::Close()

The PeerConnection::Close() function is at the heart of managing the lifecycle of a peer connection in WebRTC. This native function is responsible for the orderly shutdown of a connection. However, its behavior can be complex, with several conditional checks and sub-procedures that cater to different stages of the connection’s lifecycle.

In essence, this function first checks if the connection is already closed. If it’s not, it proceeds to update connection states, signalling its closure to any observers, and stops all transceivers. It also ensures all asynchronous stats requests are completed before destroying the transport controller. It then releases related resources like the voice/video/data channels, event log, and others.

This is the WebRTC Native Function, sourced from the active branches of Chromium’s WebRTC m115 branch-head: branch-heads/5790

void PeerConnection::Close() {
  RTC_DCHECK_RUN_ON(signaling_thread());
  TRACE_EVENT0("webrtc", "PeerConnection::Close");

RTC_LOG_THREAD_BLOCK_COUNT();

if (IsClosed()) {
return;
}
// Update stats here so that we have the most recent stats for tracks and
// streams before the channels are closed.
legacy_stats_->UpdateStats(kStatsOutputLevelStandard);

ice_connection_state_ = PeerConnectionInterface::kIceConnectionClosed;
Observer()->OnIceConnectionChange(ice_connection_state_);
standardized_ice_connection_state_ =
PeerConnectionInterface::IceConnectionState::kIceConnectionClosed;
connection_state_ = PeerConnectionInterface::PeerConnectionState::kClosed;
Observer()->OnConnectionChange(connection_state_);

sdp_handler_->Close();

NoteUsageEvent(UsageEvent::CLOSE_CALLED);

if (ConfiguredForMedia()) {
for (const auto& transceiver : rtp_manager()->transceivers()->List()) {
transceiver->internal()->SetPeerConnectionClosed();
if (!transceiver->stopped())
transceiver->StopInternal();
}
}
// Ensure that all asynchronous stats requests are completed before destroying
// the transport controller below.
if (stats_collector_) {
stats_collector_->WaitForPendingRequest();
}

// Don’t destroy BaseChannels until after stats has been cleaned up so that
// the last stats request can still read from the channels.
sdp_handler_->DestroyAllChannels();

// The event log is used in the transport controller, which must be outlived
// by the former. CreateOffer by the peer connection is implemented
// asynchronously and if the peer connection is closed without resetting the
// WebRTC session description factory, the session description factory would
// call the transport controller.
sdp_handler_->ResetSessionDescFactory();
if (ConfiguredForMedia()) {
rtp_manager_->Close();
}

network_thread()->BlockingCall([this] {
// Data channels will already have been unset via the DestroyAllChannels()
// call above, which triggers a call to TeardownDataChannelTransport_n().
// TODO(tommi): ^^ That’s not exactly optimal since this is yet another
// blocking hop to the network thread during Close(). Further still, the
// voice/video/data channels will be cleared on the worker thread.
RTC_DCHECK_RUN_ON(network_thread());
transport_controller_.reset();
port_allocator_->DiscardCandidatePool();
if (network_thread_safety_) {
network_thread_safety_->SetNotAlive();
}
});

worker_thread()->BlockingCall([this] {
RTC_DCHECK_RUN_ON(worker_thread());
worker_thread_safety_->SetNotAlive();
call_.reset();
// The event log must outlive call (and any other object that uses it).
event_log_.reset();
});
ReportUsagePattern();
// The .h file says that observer can be discarded after close() returns.
// Make sure this is true.
observer_ = nullptr;

// Signal shutdown to the sdp handler. This invalidates weak pointers for
// internal pending callbacks.
sdp_handler_->PrepareForShutdown();
}

The PeerConnection::Close() function is responsible for properly closing a WebRTC PeerConnection and cleaning up all related resources. Let’s break down what’s happening in this method:

  1. RTC_DCHECK_RUN_ON(signaling_thread());: This line checks to ensure that the Close() method is being called from the signaling thread. The signaling thread is used for operations that change the state of the PeerConnection, such as handling SDP offers and answers, ICE candidates, and closing the connection.
  2. if (IsClosed()) { return; }: This line checks if the PeerConnection is already closed. If so, the method immediately returns because there’s no work to do.
  3. legacy_stats_->UpdateStats(kStatsOutputLevelStandard);: This line updates the statistics for the PeerConnection before it is closed.
  4. The next few lines set the ICE connection state and the standard connection state to “closed”, and call the OnIceConnectionChange and OnConnectionChange methods on the observer (which could be an application that’s using the WebRTC library). This notifies the observer that the connection is now closed.
  5. sdp_handler_->Close();: This line closes the SDP (Session Description Protocol) handler, which is responsible for handling the SDP offers and answers that are part of the WebRTC handshake process.
  6. The next block of code stops all active transceivers if media is configured for the connection. A transceiver is a combination of a sender and a receiver for media (audio or video) data.
  7. The next few lines clean up various resources associated with the PeerConnection, such as the transport controller (which handles the actual network transports used for the connection), the port allocator (which is used to find local and remote ports for the connection), and any ongoing statistics collections.
  8. The call to observer_ = nullptr; removes the reference to the observer, as it’s no longer needed after the connection is closed.
  9. Finally, the method calls sdp_handler_->PrepareForShutdown();, which prepares the SDP handler for shutdown by invalidating any weak pointers for internal pending callbacks.
Closing PeerConnections: Best Practices for Android & iOS

Here are some general guidelines to follow when closing peer connections, with examples tailored for Android and iOS platforms.

Android
Single PeerConnection Closure
// Assuming peerConnection is a PeerConnection object
fun disconnectPeer(peerConnection: PeerConnection) {
    // Check the connection state
    if (peerConnection.connectionState() == PeerConnection.PeerConnectionState.CONNECTED ||
        peerConnection.connectionState() == PeerConnection.PeerConnectionState.FAILED) {
        // Close each track
        peerConnection.localStreams.forEach { mediaStream ->
            mediaStream.videoTracks.forEach { it.setEnabled(false) }
            mediaStream.audioTracks.forEach { it.setEnabled(false) }
        }

// Close the connection
peerConnection.close()
}

// Nullify the reference
peerConnection = null
}

Mesh Calling PeerConnection Closure
// Assuming peerConnections is a list of PeerConnection objects
fun disconnectPeers(peerConnections: MutableList<PeerConnection>) {
    peerConnections.forEach { peerConnection ->
        // Check the connection state
        if (peerConnection.connectionState() == PeerConnection.PeerConnectionState.CONNECTED ||
            peerConnection.connectionState() == PeerConnection.PeerConnectionState.FAILED) {
            // Close each track
            peerConnection.localStreams.forEach { mediaStream ->
                mediaStream.videoTracks.forEach { it.setEnabled(false) }
                mediaStream.audioTracks.forEach { it.setEnabled(false) }
            }

// Close the connection
peerConnection.close()
}
}

// Clear the list
peerConnections.clear()
}

iOS
Single PeerConnection Closure
// Assuming peerConnection is a RTCPeerConnection object
func disconnectPeer(peerConnection: RTCPeerConnection) {
    // Check the connection state
    if (peerConnection.connectionState == .connected || 
        peerConnection.connectionState == .failed) {
        // Close each track
        peerConnection.senders.forEach { sender in
            sender.track?.isEnabled = false
        }

// Close the connection
peerConnection.close()
}

// Nullify the reference
peerConnection = nil
}

Mesh Calling PeerConnection Closure
// Assuming peerConnections is an array of RTCPeerConnection objects
func disconnectPeers(peerConnections: inout [RTCPeerConnection]) {
    peerConnections.forEach { peerConnection in
        // Check the connection state
        if (peerConnection.connectionState == .connected || 
            peerConnection.connectionState == .failed) {
            // Close each track
            peerConnection.senders.forEach { sender in
                sender.track?.isEnabled = false
            }

// Close the connection
peerConnection.close()
}
}

// Empty the array
peerConnections.removeAll()
}

The Universal Solution

Here is an improved approach for a seamless closing of PeerConnections:

// Assuming peers is an array of RTCPeerConnection objects
function disconnectPeers(peers) {
    peers.forEach(peer => {
        // Close each track
        peer.getTracks().forEach(track => {
            track.stop();
        });

// Remove all event listeners
peer.ontrack = null;
peer.onremovetrack = null;
peer.onicecandidate = null;
peer.oniceconnectionstatechange = null;
peer.onsignalingstatechange = null;

// Close the connection
peer.close();
});

// Empty the array
peers = [];
}

This approach ensures that all resources and event listeners associated with the PeerConnection are correctly closed and removed. Each track associated with the PeerConnection is individually stopped, ensuring a complete and safe disconnection. Removing event listeners aids in preventing any unanticipated triggers after the connection has been closed.

Expert Opinion

Navigating through the complexities of WebRTC can seem like a daunting task, but it’s all part of the journey. As the saying goes, “Every expert was once a beginner.” I’ve put together this guide with the intention of making that journey a little less complicated for you.

“In the realm of WebRTC, effective resource management is not just an option, it’s a necessity. Treat it like a puzzle, where each piece must fit perfectly for the whole picture to make sense.”

Through careful understanding and application of these principles, I believe you’ll be able to overcome any hurdles you encounter in the realm of peer connections. Let’s keep learning and growing together in this fascinating field!

Sam Altman Just Raised $6.6 Billion for OpenAI — But This Is Only the Beginning of the AI Arms Race 🚀

After announcing a record $6.6 billion funding round, OpenAI revealed it also has secured a $4 billion revolving credit facility from major banks. The access to over $10 billion is sorely needed as the artificial intelligence startup burns through cash seeking “to meet growing demand.” As it expands, OpenAI is also undergoing a concentration of power.

Outgoing CTO Mira Murati won’t be replaced, and technical teams will now report directly to CEO Sam Altman himself. With several key executives already gone, “a lot of the opposition to him is gone,” whispers a source from Bloomberg. It seems Sam’s vision is now uncontested.

Crux:

🚀 Sam Altman Just Raised the BIGGEST Venture Round in History — $6.6 BILLION for OpenAI! 🤯

But hold on — this might not even be the biggest round we’ll see this year or next. Sam’s just getting started, and the next raise could blow the roof off! 🏗️💥

Who’s In on This Historic Round? 🧐

The usual suspects: Tiger Global, Khosla Ventures, Microsoft, NVIDIA… all the cool kids. And Apple? Yeah, they took a hard pass. Word has it, Tim Cook said, “Eh, too spicy for me.” 🌶️

Let’s Talk Numbers

Valuation? A casual $157 BILLION. 🤑 No biggie, right? And by the way, this is no typical equity round — these are convertible notes. Yep, some financial wizardry is happening to turn OpenAI from “not-for-profit” into “let’s-make-Sam-even-richer” mode. Gotta love the lawyers. 💼💸

A Monopoly on Innovation?

Get this — OpenAI has politely asked its investors to stay out of rival AI projects. Sorry Elon and Ilya, Sam’s got you cornered for now. He’s rallying his team and partners like a seasoned pro. 👑🎤

Is $6.6 Billion the End of the Story?

Not even close. OpenAI is burning through cash faster than a Tesla in Ludicrous Mode! Rumor has it they lost a staggering $5 BILLION last year — on only $3.7B in revenue. And with ambitious plans to build actual nuclear reactors to power their future AI, Sam’s gonna need a lot more than pocket change. 💰🔥

What’s Next?

My bet? Another raise — this time pulling in sovereign wealth fund money. The AI arms race is just heating up, and OpenAI is zooming full throttle toward AGI (Artificial General Intelligence).

Will $6.6B Cut It?

Probably not. Buckle up, folks — Sam Altman’s far from done, and OpenAI is pushing the envelope on what’s possible. 🚀

Authentic Sources!

ChatGPT Code Interpret Hacks to Run C++ Code: Navigating the Sandbox

In an era where coding is as much about creativity as it is about logic, finding new ways to experiment and execute code efficiently has become a pursuit for many developers. My latest adventure with ChatGPT, a remarkable AI by OpenAI, serves as a testament to this ongoing quest. This post delves into how I navigated the constraints of a sandbox environment to run C++ code, revealing insights and tricks that might just revolutionize your coding sessions.

The Challenge: Running C++ in a Sandbox

I came across a Twitter post where someone shared their method for compiling C/C++ code as follows:

With this motivation in mind, I immediately opened the ChatGPT console and began working on C++ using a code interpreter.

The journey began in the confines of a sandbox environment — a place where direct execution of certain commands and scripts faces limitations. Here, every action is isolated, making traditional coding practices like changing directories or directly compiling code using familiar tools like g++ seem like feats of magic. I have installed g++ as follows;

The Setup: Crafting a Directory from Scratch

Here are some initial chats:

  • Initially observed, what is available in sandbox:
  • Then, I decided to check what is my working directory in sandbox:
  • My first step was to create a new playground for my C++ explorations. The command mkdir playground-usman was my incantation, conjuring a space within the digital ether where my code could live and breathe. This was not just a directory; it was the first stone laid in the foundation of my development citadel.
The Evolution: From Simple Echo to Complex Execution

With my playground ready, the next phase was to breathe life into it. I started simple, using the echo command to create a basic C++ file. However, my ambitions quickly outgrew basic operations. I sought to weave complexity and color into my code, embedding ANSI escape codes for colored output and sophisticated string operations. This wasn’t just code; it was art.

The Culmination: Compiling and Execution within the Sandbox

The climax of my adventure came with compiling and executing the C++ code. This step, seemingly mundane outside the sandbox, was a feat of ingenuity within it. I used ChatGPT to circumvent the environment’s limitations, compiling the code with g++ and executing the resultant binary.

The output was more than just a success message; it was a beacon of triumph, signaling that within the sandbox’s constraints, creativity knows no bounds.

Insights and Reflections

This journey was not just about running C++ code; it was about pushing boundaries, about understanding the potential of AI-assisted coding.

“ChatGPT proved to be an invaluable ally, interpreting commands and executing actions that seemed initially out of reach.”

Conclusion: A New Frontier for Developers

As I reflect on this adventure, I see it as a glimpse into the future of coding — a future where AI tools like ChatGPT not only assist in coding but also enable developers to navigate and transcend traditional limitations. This experience has not just been about overcoming technical hurdles; it’s been a reaffirmation of the creative spirit that drives all developers.

Revolutionizing AR/VR Communication: The Role of the Unity WebRTC SDK​

Imagine a world where distance is no barrier to immersive, collaborative experiences. Now, stop imagining, because with the Unity WebRTC SDK, that world is becoming a reality. The famous Greek philosopher Epictetus once said, “Progress is not achieved by luck or accident, but by working on yourself daily.” This statement has become my mantra as I endeavored to create the Unity WebRTC SDK for AR/VR communication.

For three grueling weeks, I dedicated myself to the task of developing this innovation, often putting in sixteen-hour days. The outcome? An SDK that bridges the gap between WebRTC and Unity to facilitate live audio-video calling between AR/VR glasses.

AR Remote Video Calling Experience
Unity and WebRTC: A Powerful Union

WebRTC is an open-source project that provides real-time communication capabilities to web browsers and mobile applications. This technology, combined with Unity, a robust platform for creating multi-platform games and interactive experiences, is the foundation of the Unity WebRTC SDK. This SDK will soon be available on the Unity asset store, providing a tool that supports AR/VR audio-video calling for any WebRTC-supporting browser.

Remote Call between AR/VR and Browser

You can also visit my LinkedIn posts to get more exciting details:

Impact on the AR/VR Landscape

This technological advancement is set to revolutionize various sectors, particularly in the realm of remote assistance. With the Unity WebRTC SDK, AR/VR communication solutions can become a staple in healthcare, manufacturing, education, and more, fostering superior collaboration and operational efficiency.

Consider this: the global augmented reality market size was valued at USD 25.33 billion in 2021 and is projected to grow to a staggering USD 781.17 billion by 2030, according to a report by SkyQuestt.

Similarly, you can also visit these links:

In line with these predictions, my Unity WebRTC SDK is set to play a significant role in supporting this growth by providing robust communication tools for the expanding AR/VR industry.

For example, Audi has integrated AR/VR glasses in their new trucks offering drivers a unique navigational experience.

This January Audi presented the the Audi activesphere — a concept car featuring a new AR-based driver UI designed and created with Magic Leap 2. The Audi Design Studio, which conceived and designed the fourth of their sphere concept cars, focused on advancing Audi’s vision of premium mobility forward. Audi dimensions, the new UI concept, uses augmented reality to fundamentally transform the driver experience.

“Removing physical displays and touchscreens creates a new experience space for the passengers while using the power of mixed reality opens up new possibilities to intuitively integrate them into the mobility ecosystem and it goes even beyond the car,” said Jan Pflueger, Creative Technologist at Audi Design.

Audi VR GIF

With my SDK, the possibility for live, real-time communication between these vehicles becomes a reality, further enhancing the driving experience and improving road safety.

The Future is Open Source

I strongly believe in the power of collective innovation. That’s why I’m planning to open-source the Unity WebRTC SDK soon. Open-sourcing the SDK will allow developers worldwide to contribute to its growth, driving the future of AR/VR communication technology.

Concluding Thoughts

The development of the Unity WebRTC SDK underlines the transformative power of innovative thinking and hard work. By seamlessly merging the realms of AR/VR and WebRTC, this SDK is not just a new tool in the box — it’s a revolutionary stride towards a future where immersive, interactive experiences are commonplace, and distance is no obstacle to collaboration.

The Future of Remote Work: Unleashing AI-Powered Communication with WebRTC and Machine Learning

4 years ago, I decided to take a hiatus from Machine Learning, a field that had been a part of my life for an extensive period. The decision was not easy, but it provided me with a different perspective and an opportunity to explore the vast world of technology outside the realm of Machine Learning.

In the interim, I stumbled upon the dynamic technology of WebRTC, which was rapidly transforming the landscape of real-time communication. The more I delved into it, the more I became fascinated by its potential and its pivotal role in powering remote collaboration.

Recently, an idea sparked in my mind — a notion to merge these two technological titans, WebRTC and Machine Learning. As I began to understand the capabilities of this merger, I became more and more excited. I realized that I was not merely bringing Machine Learning back into my life but was reintroducing it in a way that could create significant value in the realm of real-time media and the broader business industry.

Today, I stand at the junction of my technological journey, where the road ahead is teeming with potential. I’m eager to share the insights and ideas that this fusion of WebRTC and Machine Learning has inspired in me, and I hope to contribute meaningfully to the evolution of remote collaboration.

Fasten your seat belts as we dive deep into this exciting journey of marrying real-time communication with the power of AI. Let’s explore the endless possibilities together!

Remote Collaborations:

The landscape of remote collaboration has witnessed a tectonic shift over the past decade, thanks to the groundbreaking advancements in WebRTC and Machine Learning. WebRTC (Web Real-Time Communication) has emerged as a trailblazer, powering instantaneous and high-quality P2P communication. Simultaneously, Machine Learning, an AI subset, has marked its omnipresence across multiple sectors, promising a future enriched by intelligent digital experiences. The amalgamation of these technologies offers a vast array of possibilities that could redefine remote collaboration. Let’s explore this intriguing confluence in more depth.

The Power Duo: WebRTC and Machine Learning

WebRTC, with its seamless P2P communication capabilities, has proved to be an indispensable asset for video conferencing and live streaming applications. Machine Learning, on the other hand, has been transforming the world by learning from data and recognizing patterns — creating solutions ranging from customized email responses to autonomous vehicles. When applied to communication, Machine Learning can significantly enhance user experience and productivity.

Envisioning The Integration: Machine Learning Infused WebRTC

Well, I am going to give you a simple start to the ideology using Pseudo Code examples to explore the infusion. In the next series, I will be coming up with the required ML models to do the magic for us.

Envisage your video conferencing tool equipped with features that learn and adapt to provide an improved, personalized experience. This scenario exemplifies the power of integration between WebRTC and Machine Learning. Let’s examine a few exciting use cases this alliance could potentially offer:

  1. AI-aided Video and Audio Quality
  2. Real-time Translations and Transcriptions
  3. Intelligent Noise Cancellation
  4. Sentimental Analysis

AI-aided Video and Audio Quality Enhancements: Machine Learning algorithms can assist WebRTC in analyzing network conditions and fine-tuning the video and audio quality in real-time.

# Pseudo Code
import webrtcvad # voice_activity_detection
import some_ml_model

class WebRTCAIEnhancement:
def __init__(self):
self.vad = webrtcvad.Vad()
self.ml_model = some_ml_model.load_model(‘model_path’)

def optimize_stream(self, stream_data):
# Use WebRTC to detect speech frames
is_speech = self.vad.is_speech(stream_data, sample_rate)

# Use ML model to enhance audio quality
if is_speech:
enhanced_data = self.ml_model.predict(stream_data)
return enhanced_data

Real-Time Translations and Transcriptions: Integrating Machine Learning with WebRTC can enable real-time translations, breaking language barriers, and facilitating more inclusive conversations.

# Pseudo Code
import webrtcvad # voice_activity_detection
from some_speech_recognition_library import SpeechRecognition
from some_translation_library import Translator

class RealTimeTranslation:
def __init__(self):
self.vad = webrtcvad.Vad()
self.recognizer = SpeechRecognition()
self.translator = Translator()

def translate(self, stream_data, target_language):
# Use WebRTC to detect speech frames
is_speech = self.vad.is_speech(stream_data, sample_rate)

# Transcribe and translate
if is_speech:
text = self.recognizer.transcribe(stream_data)
translated_text = self.translator.translate(text, target_language)
return translated_text

Intelligent Noise Cancellation: Machine Learning algorithms can work alongside WebRTC to differentiate human speech from background noise, enhancing audio clarity.

# Pseudo Code
import webrtcvad # voice_activity_detection
import some_noise_cancellation_model

class WebRTCNoiseCancellation:
def __init__(self):
self.vad = webrtcvad.Vad()
self.ml_model = some_noise_cancellation_model.load_model(‘model_path’)

def remove_noise(self, stream_data):
# Use WebRTC to detect speech frames
is_speech = self.vad.is_speech(stream_data, sample_rate)

# Use ML model to remove noise
if is_speech:
noise_free_audio = self.ml_model.predict(stream_data)
return noise_free_audio

Sentiment Analysis: Machine Learning can assess the emotional state of participants during a video call, providing valuable feedback and promoting better communication.

# Pseudo Code
import webrtcvad # voice_activity_detection
from some_sentiment_analysis_library import SentimentAnalyzer

class WebRTCSentimentAnalysis:
def __init__(self):
self.vad = webrtcvad.Vad()
self.analyzer = SentimentAnalyzer()

def analyze_emotion(self, stream_data):
# Use WebRTC to detect speech frames
is_speech = self.vad.is_speech(stream_data, sample_rate)

# Use ML model to analyze sentiment
if is_speech:
text = self.recognizer.transcribe(stream_data)
emotion = self.analyzer.analyze(text)
return emotion

Navigating the Future of AI-Enhanced Communication

The fusion of WebRTC and Machine Learning can profoundly influence remote collaboration, driving the future toward AI-enhanced communication. With potential applications ranging from intelligent video streaming to smart virtual meeting rooms, the possibilities are boundless. As these technologies evolve, they promise a future of remote work that’s not just convenient, but also smart, personalized, and efficient.

Conclusion

In an era where digital transformation is the norm rather than the exception, the convergence of WebRTC and Machine Learning stands as a beacon of innovation. This blend is a prime example of how technology can transform our work and personal lives, making communication not just faster, but smarter, more efficient, and personalized.

The future of remote work looks increasingly interconnected, with AI-enhanced video streaming and intelligent virtual meeting rooms leading the way. As the boundaries of remote collaboration continue to expand, it’s essential to embrace these advancements and explore their potential to the fullest.

Expert Opinion

As a tech enthusiast and an ardent observer of digital evolution, I believe the symbiosis of WebRTC and Machine Learning will dramatically shape the future of remote collaboration. The dynamism of WebRTC, coupled with the data-driven insights of Machine Learning, offers a robust solution that goes beyond mere communication — it learns, adapts, and enhances.

This integration not only sets the stage for more efficient communication but also promises more inclusive, accessible, and personalized user experiences. As we move forward, organizations and individuals who harness the potential of this synergistic technology will undoubtedly lead the pack in the evolving landscape of remote work.

The fusion of WebRTC and Machine Learning is more than just a technological leap. It’s a stride toward a more connected world, where technology doesn’t just facilitate communication; it elevates it.

LiveKit Can Handle Millions of Concurrent Calls — Crazy!

In today’s hyper-connected world, real-time communication is the backbone of our daily interactions. From virtual meetings and online gaming to live streaming events, the demand for seamless, high-quality audio and video communication is skyrocketing. But have you ever wondered what it takes to handle millions of concurrent calls without a hiccup? Enter LiveKit, a real-time audio and video platform that’s redefining scalability with its sophisticated cloud infrastructure.

It is LiveKit that has come into the ocean of the extremely crowded WebRTC and VoIP industry and pulled off a miracle.

David Zhao is the CTO and Co-Founder of LiveKit. Motivation behind the article.

Comment from CoFounder of Livekit
The Real-Time Communication Challenge

Handling real-time communication at scale is no small feat. Traditional systems often buckle under the pressure of high concurrency, leading to dropped calls, lagging videos, and frustrated users. The challenge lies in managing a vast number of simultaneous connections while maintaining low latency, high reliability, and excellent quality of service.

LiveKit’s Cloud Architecture Inspection

Hey, we’re not talking about Zoom, Microsoft Teams, Google Meet, Cisco Webex, or Voov. We’re discussing something truly innovative and exciting. LiveKit tackles the challenges of real-time communication head-on with a cloud-native architecture designed for scalability, reliability, and performance. Let’s dive into the key components that make LiveKit’s infrastructure capable of handling millions of concurrent calls.

Fun game by Cofounders of Livekit
1. Selective Forwarding Unit (SFU) at the Core

At the heart of LiveKit’s architecture is the Selective Forwarding Unit (SFU). Unlike traditional Multipoint Control Units (MCUs) that mix media streams — adding latency and consuming significant resources — SFUs act as intelligent routers that forward media streams to participants based on their subscriptions.

  • Scalability: SFUs efficiently manage bandwidth by sending only the necessary streams to each participant.
  • Low Latency: By avoiding media transcoding, SFUs keep latency minimal.
  • Resource Efficiency: Reduced computational overhead allows for better resource utilization.
2. Distributed Mesh Network

LiveKit employs a distributed mesh network to ensure that no single point becomes a bottleneck.

  • Horizontal Scalability: New nodes can be added on-demand to handle increased traffic seamlessly.
  • Fault Tolerance: The system can withstand node failures without impacting the overall service.
  • Geographical Distribution: Deploy nodes closer to users globally to reduce latency.
3. Microservices Architecture

Breaking down the application into microservices allows for independent development, deployment, and scaling.

  • Modularity: Each service focuses on a specific function, improving maintainability.
  • Scalability: Scale individual services based on demand.
  • Continuous Deployment: Faster updates and rollbacks without affecting the entire system.
4. Kubernetes Orchestration

LiveKit leverages Kubernetes for container orchestration, ensuring efficient management of resources.

  • Automated Deployment: Streamlines the deployment process across multiple environments.
  • Self-Healing: Automatically restarts failed containers and reschedules workloads.
  • Load Balancing: Distributes network traffic to maintain service stability.
5. Cloud-Agnostic Design

LiveKit’s infrastructure is designed to be cloud-agnostic, allowing it to run on any major cloud provider like AWS, Azure, GCP, Tencent Cloud, or even on-premises servers.

  • Flexibility: Avoid vendor lock-in and choose the best cloud services for your needs.
  • Hybrid Deployments: Combine cloud and on-premises resources for optimal performance and compliance.
  • Cost Optimization: Leverage various pricing models to manage expenses effectively.
Aforementioned pieces together
Practical Insights into LiveKit’s Infrastructure
Auto-Scaling Capabilities
  • Horizontal Scaling: Automatically add more instances when demand increases.
  • Vertical Scaling: Adjust resources like CPU and memory on existing instances.
  • Real-Time Monitoring: Use metrics to trigger scaling events proactively.
Global Presence with Edge Nodes

Deploying edge nodes in multiple regions brings services closer to users.

  • Reduced Latency: Improves user experience by minimizing data travel time.
  • Regional Compliance: Adhere to data sovereignty laws by keeping data within specific regions.
  • Load Distribution: Balance traffic across multiple nodes to prevent overloads.
Advanced Session Management
  • Room-Based Architecture: Organize sessions into rooms for better management.
  • Dynamic Stream Routing: Adjust stream quality and routing based on network conditions.
  • Participant Capabilities: Tailor media streams to match each participant’s device and network capabilities.
Security and Compliance
  • End-to-End Encryption: Protects media streams from interception.
  • Secure Authentication: Uses tokens and secure protocols for user authentication.
  • Compliance Standards: Aligns with GDPR, HIPAA, and other regulations as needed.
Developer-Friendly APIs
  • Comprehensive SDKs: Available for multiple platforms, simplifying integration.
  • Customization: APIs allow for extensive customization to fit specific application needs.
  • Documentation and Support: Detailed guides and active community support streamline development.
Integrating LiveKit with DevOps Practices
Infrastructure as Code (IaC)
  • Terraform and Helm Charts: Use IaC tools for consistent and repeatable deployments.
  • Version Control: Track infrastructure changes alongside application code.
  • Automation: Reduce manual errors and accelerate deployment cycles.
Continuous Integration/Continuous Deployment (CI/CD)
  • Automated Testing: Ensure code quality with unit and integration tests.
  • Pipeline Orchestration: Tools like Jenkins, GitLab CI/CD, or GitHub Actions automate the build and deployment process.
  • Rollback Strategies: Implement blue-green or canary deployments for safer releases.
Monitoring and Logging
  • Observability Tools: Integrate with Prometheus, Grafana, or Datadog for real-time monitoring.
  • Alerting Systems: Set up alerts for performance issues or failures.
  • Log Aggregation: Centralize logs for easier troubleshooting and analysis.

To get more in-depth understanding, please visit:

A Practical Overview of Other Open-Source Media Servers

While LiveKit offers a robust and scalable solution, other open-source media servers also have unique strengths. Having personally engineered and set up cloud infrastructures for these servers, here’s what each brings to the table:

Ant Media Server
  • Key Strengths: Ultra-low latency streaming (as low as 0.5 seconds) and adaptive bitrate support.
  • Noteworthy Aspects: Backed by the most lovely and amazing team, Ant Media Server is a leading streaming provider known for exceptional customer support and innovation.
Janus Media Gateway

Janus Media server has super legacy in real-time communications.

  • Key Strengths: Highly modular with a plugin-based architecture.
  • Best For: Custom solutions requiring specific features like SIP gateways or data channels
Kurento Media Server

I personally call ‘ Kurento Media Server ‘ as the mother of all the media servers.

  • Key Strengths: Advanced media processing, including real-time filters and computer vision capabilities.
  • Best For: Applications needing media transformations like augmented reality or special effects.
MediaMTX (formerly rtsp-simple-server)

MediaMTX acts as ‘Media Proxy’ or ‘Media Router ’ because of it’s ready-to-use and zero-dependency real-time capabilities.

  • Key Strengths: Lightweight and straightforward to deploy.
  • Best For: Simple RTSP or RTMP streaming needs with minimal overhead.

Sora

WebRTC SFU Sora is my crush, build by Japanese team. They have reached their 13th fiscal year 2024.10.01.

  • Key Strengths: Enterprise-level scalability and reliability with focus on Japanese market needs.
  • Best For: Large-scale deployments requiring consistent performance and support.
  • 2024/07/16 Seven Switch Co., Ltd. has adopted Sora.
mediasoup

mediasoup is also a perfect choice for building multi-party video conferencing and real-time streaming apps.

  • Key Strengths: High-performance SFU written in Node.js.
  • Best For: Developers seeking flexibility and deep control over media handling.
Why LiveKit Stands Out

While each media server has its merits, LiveKit distinguishes itself through its cloud-native, scalable architecture designed for modern applications.

  • Scalability: Seamlessly handle millions of concurrent calls without compromising performance.
  • Developer Experience: Rich SDKs and APIs make integration straightforward.
  • Cloud Integration: Easily deploy across various cloud providers with Kubernetes orchestration.
  • Active Community: An open-source project with a growing community contributing to its evolution.
AI Generated Livekit Description

Moreover, the ability to handle such massive concurrency with a focus on maintaining low latency and high quality sets LiveKit apart from the competition.

  • OpenAI and LiveKit partner to turn Advanced Voice into an API
Livekit’s Founder’s Story
Potential Improvements and Future Directions

Even with its impressive architecture, LiveKit has room to grow:

Edge Computing Integration

  • Opportunity: Deploy SFUs at the edge to bring processing closer to users.
  • Benefit: Further reduce latency and improve user experience in geographically diverse deployments.

AI-Powered Network Optimization

  • Opportunity: Implement AI algorithms to predict network congestion and adjust streams proactively.
  • Benefit: Enhance quality of service by adapting to network conditions in real-time.

Enhanced Security Measures

  • Opportunity: Introduce features like zero-trust security models and advanced encryption standards.
  • Benefit: Strengthen security posture, making it suitable for sensitive applications.

Serverless Architecture Exploration

  • Opportunity: Investigate serverless computing for certain components to optimize resource usage.
  • Benefit: Reduce operational costs and scale automatically based on demand.
Impact on WebRTC and the LiveKit Community

LiveKit is not just a media server; it’s a catalyst for innovation in the WebRTC space.

  • Simplifying Complexity: Abstracts the complexities of real-time communication, allowing developers to focus on building features.
  • Fostering Collaboration: An open-source project that encourages community contributions and shared learning.
  • Driving Industry Standards: Influences best practices and standards within the WebRTC community through its innovative approaches.
Conclusion

Handling millions of concurrent calls is no longer an insurmountable challenge — it’s a reality with LiveKit’s advanced cloud architecture. By combining SFU technology, distributed networking, microservices, and Kubernetes orchestration, LiveKit offers a scalable, reliable, and efficient platform for real-time communication.

Having more than six years of experience in the WebRTC and VoIP industry, and worked extensively with various media servers, I can attest that LiveKit’s cloud infrastructure is truly revolutionary. While other platforms like Ant Media Server — backed by their amazing team — and Sora offer excellent solutions, LiveKit’s emphasis on scalability and developer experience makes it a standout choice.

I can confidently say that now is the perfect time to jump in with the right open-source module. LiveKit, with its rapid development — much like Elon Musk’s rockets soaring into space — is revolutionizing real-time communication. Its cutting-edge features and the pace at which it’s evolving make it an exciting platform to watch and be a part of.

Stay tuned for more updates!

Venturing Into the Uncharted: My First IETF Meeting Experience

Stepping Into a New Chapter

I am thrilled to announce an exciting new chapter in my professional journey: my first-ever participation in the Internet Engineering Task Force (IETF) meeting from July 22nd to 28th. This global event marks a significant milestone in my journey as a WebRTC Engineer.

“Every new beginning comes from some other beginning’s end.” — Seneca

IETF Event
Who Am I?

Often fondly referred to as ‘Mr. WebRTC,’ my experiences range from real-time audio and video communication to game streaming. My noteworthy contributions to technology include a U.S. patent and pivotal software consultation for NHS England.

The IETF: An Overview

The IETF is an open international community of network designers, operators, vendors, and researchers. They share the common interest of evolving the Internet architecture and ensuring its smooth operation.

IETF
My Journey to IETF

Diving into the deep end of IETF, one can unlock fresh insights, connect with industry leaders, and contribute to the collective knowledge of the field. My preparation for this event involved more than a month of studying IETF’s various aspects, understanding its importance, and how it could shape my professional standing.

The Titans of IETF

Backed by industry giants like Cisco, Meta, and Nokia, the IETF stands as a monument to innovation and progress. The intrigue of the IETF extends to other heavyweights like IBM, Google, and Ericsson, all working towards a shared vision for the future of the Internet.

“Coming together is a beginning, staying together is progress, and working together is success.” — Henry Ford

An Exciting Melting Pot

The IETF is a fascinating blend of contributors where thought leaders from world-renowned firms stand shoulder to shoulder with independent professionals and researchers, fostering a hotbed of ideas.

Seizing the Opportunity

The philosopher Seneca once said, “Luck is what happens when preparation meets opportunity.” The meticulous groundwork laid over the past month is about to meet the incredible opportunity of attending the IETF meeting.

The Potential of Participation

Participating in IETF meetings and contributing to its Working Groups can significantly boost one’s professional profile, providing worldwide recognition and enriching one’s expertise in Internet standards and protocols.

“Innovation distinguishes between a leader and a follower.” — Steve Jobs

I’m thrilled to announce that I’m working on a groundbreaking research draft for the upcoming IETF meeting. The topic, “Optimization of NAT Traversal for WebRTC Connections,” aims to revolutionize the way we establish WebRTC connections by introducing an innovative predictive model. This breakthrough not only enhances the speed but also the efficiency of NAT traversal, paving the way for swifter, more robust real-time communication. So, stay tuned for this cutting-edge development in the world of WebRTC!

A Word for the Newbies

Every expert was once a beginner, and platforms like IETF offer an ideal place to learn, grow, and become industry leaders. As long as you’re willing to put in the hard work and the hours, there’s no telling where this journey can take you.

The Future Awaits

I invite you all to follow my journey through this IETF meeting. I aim to bring back and share fresh perspectives, ground-breaking ideas, and hope to inspire more professionals to partake in such impactful initiatives.

“If I have seen further, it is by standing on the shoulders of giants.” — Isaac Newton

The World’s Fastest Voice Agent with AI, WebRTC, Whisper, and Latency Comparisons

Basic View of Agent

As a software inventor, I’m always exploring new ways to make existing systems faster, more intuitive, and creative. My journey took an exciting turn when I came across a post on Y Combinator’s community by Cerebrium.ai. Their work on reducing latency in voice bots to 500ms caught my attention, and I wondered — could I push the limits even further, perhaps even below 300ms?

Demo Video: Lowest Latency Customer Care Agent

The Spark: A Post That Changed Everything

Cerebrium’s post highlighted the challenges of building AI that can respond as quickly as humans in conversation at the Y-Combinator Community.

https://news.ycombinator.com/item?id=40805010

They shared their metrics and a demo of a voice bot capable of 500ms voice-to-voice response time. Their approach emphasized hosting transcription, LLM inference, and voice generation in one place to minimize latency.

Inspired by their work, I sought to see if I could optimize a voice customer care agent even further — despite not having the resources or a team like Cerebrium’s. This was a challenge driven by curiosity and the desire to see what I could accomplish independently.

Redesigning the Idea: A Step-by-Step Journey of Latency Optimization

I began by breaking down each component of the voice agent’s latency and identifying areas for improvement. The journey wasn’t easy, but step by step, I made progress.

1. Mic Input Optimization

Challenge: The initial latency at the microphone input was a significant bottleneck at 40ms.

  • Solution: I integrated WebRTC for faster input processing, cutting the latency in half.
Mic Latency Comparison

2. Opus Encoding/Decoding

Challenge: Opus encoding and decoding were each taking 30ms, adding up to a noticeable delay.

Solution: I optimized the codec processing, reducing the latency for both stages.

Opus Latency Comparison

3. Jitter Buffer Optimization

Challenge: The jitter buffer was causing a delay of 40ms.

  • Solution: By reducing the buffer time, I brought it down to a minimal 10ms.
Jitter Latency Comparison

4. Transcription & Endpoints

Challenge: Transcription and endpointing were the biggest hurdles, with a latency of 200ms.

Solution: I trained a Whisper model to handle 10ms audio chunks, drastically improving this stage.

Transcription Latency Comparison

5. Proofreading Layer

Challenge: To ensure clarity and accuracy in communication, I needed a dynamic proofreading mechanism.

  • Solution: I added a proofreading layer that refines the text in real time with minimal latency.
Proofread Latency Comparison

6. LLM and TTS Optimization

Challenge: LLM and TTS were taking significant time, at 100ms and 80ms, respectively.

Solution: I fine-tuned both processes to cut their latency in half.

LLM Latency Comparison

The Result: A Sub-300ms Latency Voice Agent

After addressing each of these challenges, I’m proud to share that I’ve successfully reduced the total latency to 239ms. This achievement is not just a technical milestone but a testament to what can be accomplished with passion, persistence, and a deep understanding of AI and real-time systems.

Latency Comparison

AI’s Role in Revolutionizing Customer Care

AI is increasingly becoming a game-changer in customer care, offering speed, consistency, and personalized experiences. Voice customer care agents are at the forefront of this revolution, providing 24/7 support with the ability to handle complex queries efficiently. As AI continues to evolve, these agents will become even more adept at understanding and responding to customers in real time, making interactions smoother and more human-like.

My journey to optimize latency is a small but significant step in this larger movement. By focusing on the details and pushing the boundaries of what’s possible, I’ve been able to create a voice agent that operates with sub-300ms latency — a major improvement over existing systems.

Conclusion: The Power of Innovation

This project has shown me that innovation isn’t just about having the right resources — it’s about having the right mindset. With passion and persistence, I’ve been able to achieve something remarkable, reducing latency to 239ms and enhancing the performance of AI-driven customer care agents.

I’m excited to continue exploring this field, and I look forward to sharing more about the architecture and performance improvements. Thank you to everyone who has supported me on this journey.

Stay tuned for more updates!

OBS Studio and WebRTC Development and Real-time Video Streaming | World’s fastest and real-time streaming solution

Some WebRTC geeks mostly come up with a couple of questions;

  1. How to stream video using OBS Studio over WebRTC?
  2. Is it possible to send Audio Video Packets to WebRTC endpoints using OBS Studio?

Well, this is possible and let’s have a short demo and explanation of such kind of research work.

Last year, I have spend a considerable amount of research and development to incorporate WebRTC in other FFMPEG for real-time Video Streaming. The research work involved in developing FFMPEG and WebRTC together was a big challenge for me. The main reason was code stack compatibility as FFMPEG is being developed in C language whereas WebRTC is continuously being developed using C++/C/Assembly directly to manage resource-intensive concurrent tasks. Another big challenge was the Audio, Video Data Compatibility between FFMPEG and WebRTC.

Alhamdulillah, After spending 3, or 4 months of time last year, I was able to stream H264 encoded video packets from FFMPEG over WebRTC endpoints in a P2P way. After that, I worked on the scalability of FFMPEG WebRTC Solution using AntMedia Server and I was able to make that happen as well. You can find a demo implementation of FFMPEG WebRTC Streamer here; https://github.com/mail2chromium/ffmpeg-webrtc-streamer

Today is a special day for me as I have been honored with the title of “Mr. WebRTC” from AntMedia Cofounder Mr. Ahmet Oğuz Mermerkaya. This recognition has encouraged me to push my limits and strive for more success in my career. Despite the loss of My Mother just two months ago, I feel that she is with me in spirit and all of my achievements are a testament to the wonderful upbringing she provided me. Her teachings have instilled in me a strong mindset that has driven me to where I am today.

https://www.linkedin.com/posts/ahmetoguzmermerkaya_broadcast-community-webrtc-activity-7026858691370856448-HP7b?utm_source=share&utm_medium=member_desktop

Two big giants of the media industry together:

This was the main motivation for me to try another multi-media giant project which is OBS Studio. When I started FFMPEG WebRTC together at that time, OBS-Studio was also missing WebRTC Video Streaming. As the motivation was great, so I didn’t miss time to incorporate WebRTC inside OBS Studio. Well, the main challenges with OBS Studio and WebRTC were again the same such;

– Different code stack, implementation standards, and style guides

– Audio Video Data Compatibility and

– Getting access to audio-video streams from OBS Studio

– Assigning captured audio, and video streams from OBS-Studio to WebRTC.

– Handling different timestamps and audio, and video bit rates for both OBS-Studio and WebRTC.

– Audio, Video processing, and streaming to other WebRTC Endpoints.

– Handling VP8, VP9, H264, and H265 Video Codecs compatibility for both OBS-Studio and WebRTC.

Keeping all the above research works in the loop, I made it without wasting any time.

A simple snapshot to show OBS + WebRTC Streaming to Custom WebRTC Player.

OBS + WebRTC Streaming to Custom WebRTC Player

Explanation of WebRTC usage inside OBS-Studio:

  1. Setting up OBS Studio: In OBS Studio, you will need to set up your audio and video sources, such as your microphone and webcam, as well as configure the settings for encoding and compressing the video.
  2. Setting up WebRTC: On the WebRTC side, you will need to create a peer connection and add the necessary JavaScript libraries to your website or web application or set up an android/iOS webrtc endpoint to get obs-studio real-time streams.
  3. Capturing OBS Studio output: To capture the audio and video packets from OBS Studio, you can use the OBS Studio API to access the output of the stream. You can use the obs_output_t struct to receive the video and audio frames.
  4. Send packets to WebRTC pipeline: Once you have captured the audio and video packets from OBS Studio, you can send them to the WebRTC pipeline using the RTCPeerConnection.addTrack() method. This method allows you to add a track to the peer connection, which can be either an audio or video track.
  5. Encoding and Decoding: You will need to configure the codecs that OBS Studio uses to encode the audio and video packets so that they can be decoded by WebRTC on the remote participants.
  6. Sending to the remote peer: Once the packet is sent to the WebRTC pipeline, it will be encoded and sent to the remote peer over the network.
  7. Receive and decode packets: On the other end, the packets will be received and decoded by the remote peer’s WebRTC pipeline, allowing them to display the video and hear the audio.

It’s important to note that this process is complex and requires a good understanding of both OBS Studio and WebRTC. It is also important to consider the security measures to protect the data being sent.

Access OBS Audio&Video Packets:

To get audio and video packets from an obs_output_t the object inside the OBS Studio program, you can use the obs_output_get_video_data and obs_output_get_audio_data functions, respectively. These functions return a pointer to the video or audio data, which can then be processed or written to a file.

Here is an example of how you might use these functions to get audio and video data from an output:

obs_output_t* output = obs_output_create("rtmp_output", "my_stream", nullptr, nullptr);

// …

// Get video data
video_data = obs_output_get_video_data(output);
if (video_data) {
// Process video data
// …
}

// Get audio data
audio_data = obs_output_get_audio_data(output);
if (audio_data) {
// Process audio data
// …
}

// …

obs_output_release(output);

It is important to note that the obs_output_get_video_data and obs_output_get_audio_data functions only return valid data if the output is actively streaming.

Also, make sure to call the obs_output_release(output) after you are done with it.

Pass OBS Audio&Video Packets to WebRTC Pipeline:
Once you have the video or audio frame, you can then use the webrtc::VideoFrame and webrtc::AudioFrame classes to add the frames to the OnFrame() callback of your WebRTC implementation.

Here is an example of how you might use these functions to get audio and video frames from an obs_output_t and add them to a WebRTC OnFrame() callback:

void OnFrame(webrtc::VideoFrame& frame) {
    obs_output_t* output = GetOutput();
    if (output) {
        obs_source_t* video_source = obs_output_get_source(output);
        video_frame_t* video_frame = obs_source_get_frame(video_source);
        webrtc::VideoFrame rtc_frame = ConvertVideoFrameToWebRTC(video_frame);
        frame = rtc_frame;
    }
}
void OnFrame(webrtc::AudioFrame& frame) {
    obs_output_t* output = GetOutput();
    if (output) {
        obs_source_t* audio_source = obs_output_get_source(output);
        audio_data_t audio_frame;
        obs_source_output_audio(audio_source, &audio_frame);
        webrtc::AudioFrame rtc_frame = ConvertAudioFrameToWebRTC(audio_frame);
        frame = rtc_frame;
    }
}

Note: The above code is just an example, you should implement the ‘ConvertAudioFrameToWebRTC’ and ‘ConvertVideoFrameToWebRTC’ functions, that convert obs_audio_data_t and obs_video_frame_t to webrtc::AudioFrame and webrtc::VideoFrame respectively.

Just to help you guys to kick start, I am sharing an implementation of basic OBSWebRTCBridge;

Please visit for more details on the Github repo: https://github.com/mail2chromium/OBSWebRTCBridge.git


/*Copy the RTCObsBridge.cc file -> Muhammad Usman Bashir -> 29-01-2023*/

#include <iostream>
#include <webrtc/api/peerconnectioninterface.h>
#include “OBS_API.h”

class RTCObsBridge {
public:
RTCObsBridge();
void Start();
void Stop();
private:
rtc::scoped_refptr<webrtc::PeerConnectionInterface> peer_connection_;
obs_source_t* audio_source_;
obs_source_t* video_source_;
};

RTCObsBridge::RTCObsBridge() {
// Initialize WebRTC
rtc::InitializeSSL();
webrtc::PeerConnectionFactory::Initialize();

// Create a new PeerConnectionFactory
rtc::scoped_refptr<webrtc::PeerConnectionFactoryInterface> factory = webrtc::CreatePeerConnectionFactory();

// Create a new PeerConnection
webrtc::PeerConnectionInterface::RTCConfiguration config;
peer_connection_ = factory->CreatePeerConnection(config, nullptr, nullptr, nullptr);

// Get OBS audio and video sources
audio_source_ = obs_get_source_by_name(“audio_source”);
video_source_ = obs_get_source_by_name(“video_source”);
}

void RTCObsBridge::Start() {
// Create media streams
rtc::scoped_refptr<webrtc::MediaStreamInterface> stream = peer_connection_->CreateLocalMediaStream(“stream”);
rtc::scoped_refptr<webrtc::AudioTrackInterface> audio_track = factory->CreateAudioTrack(“audio”, factory->CreateAudioSource(cricket::AudioOptions()));
rtc::scoped_refptr<webrtc::VideoTrackInterface> video_track = factory->CreateVideoTrack(“video”, factory->CreateVideoSource(cricket::VideoOptions()));

// Add audio and video tracks to the media stream
stream->AddTrack(audio_track);
stream->AddTrack(video_track);

// Add the media stream to the PeerConnection
if (!peer_connection_->AddStream(stream)) {
std::cerr << “Failed to add stream to PeerConnection” << std::endl;
return;
}

// Create an RTCOfferDescription
webrtc::PeerConnectionInterface::RTCOfferAnswerOptions options;
peer_connection_->CreateOffer(options);

// Set up callbacks for OBS audio and video data
obs_source_output_audio audio_data_callback = [&](void* data, size_t size) {
// Pass audio data to WebRTC pipeline
audio_track->AddSamples(data, size);
};
obs_source_output_video video_data_callback = [&](gs_texture* texture) {
// Pass video data to WebRTC pipeline
video_track->AddTexture(texture);
};

// Start the OBS audio and video sources
obs_source_add_output(video_source_, video_data_callback, nullptr);
obs_source_start(audio_source_);
obs_source_start(video_source_);
}

void RTCObsBridge::Stop() {
// Stop the OBS audio and video sources
obs_source_remove_output(audio_source_, audio_data_callback, nullptr);
obs_source_remove_output(video_source_, video_data_callback, nullptr);
obs_source_stop(audio_source_);
obs_source_stop(video_source_);

// Close the PeerConnection
peer_connection_->Close();

// Release the PeerConnectionFactory
webrtc::PeerConnectionFactory::Release();

// Deinitialize SSL
rtc::CleanupSSL();
}

Also, you’ll have to have the proper dependencies, and headers of WebRTC included in your OBS project.

Benefits of using OBS Studio and WebRTC together:

When used together, WebRTC and OBS can provide several benefits:

  1. Low Latency: WebRTC provides low-latency communication, which means that there is minimal delay between the time an action is performed and the time it is seen on the other end. This is important for live streaming and video conferencing applications. I have made this solution more scalable using AntMedia.

Recently, I performed a test with the AntMedia team during a community meeting and we together found out that Video Streaming using OBS & WebRTC has a latency of around 600 MS which is way ahead and fast than RTMP: You can see the below screenshot;

OBS & WebRTC Latency between OBS and WebRtc Antmedia Player=600

I have highlighted the Milliseconds on OBS and Player side using Yello Box and Red Arrow.

To validate my research and results, I have also given a demo video and some more screenshots here;

Demonstration of OBS + WebRTC
  1. Easy Integration: WebRTC is a browser-based technology, which means that it can be integrated into web applications without the need to install any software on the client side. This makes it easy to use and accessible to a wide range of users.
  2. High-quality Video: OBS provides a variety of options for encoding and compressing video, which means that the quality of the video can be adjusted to suit the needs of the application.
  3. Flexibility: OBS allows you to add multiple sources to your stream, such as game footage, webcam, and audio inputs.
  4. Cost-effective: Both WebRTC and OBS are free and open-source, which means that there are no licensing costs associated with using them.
  5. Cross-platform compatibility: WebRTC and OBS can work on multiple platforms like Windows, Linux, Mac, and mobile devices.

Supported Platforms & Media Servers:

This current implementation is a fast video streaming solution that utilizes the OBS Studio version 28.0.3 and WebRTC version m108. It works on both Windows and Linux operating systems and is compatible with AntMedia, Janus, and Kurento media servers. The solution also utilizes NDI and Nvidia components.

Expert Opinion:

Integrating WebRTC into OBS Studio can be a complex process, as it involves working with both software systems, which have different capabilities and settings. The process generally involves configuring OBS Studio to connect to a WebRTC server, setting up the WebRTC server to receive and transmit audio and video streams, and adjusting settings in both OBS Studio and the WebRTC server to ensure that the audio and video are synced and of good quality. Additionally, there may be additional steps required to secure the connection, such as setting up SSL/TLS encryption. It may require knowledge of coding, networking, and video streaming. It is also recommended to have a basic understanding of WebRTC, OBS Studio, and the underlying technologies that they use.

Overall, the combination of WebRTC and OBS can provide a powerful and flexible solution for live streaming and video conferencing applications, with low latency and high-quality video, and easy integration and low costs which can open up new opportunities for creators, businesses, and organizations to reach and engage with their audience.

Probability and Statistics from Scratch in Julia

Motivation: As part of my personal journey to gain a better understanding of Probability & Statistics, I just follow courses from the world’s best educational Systems such as i.e. MIT, and Stanford. I’ve decided to draw a complete picture of these concepts in Julia Langauge (MIT Labs). Every time, It is basically, the Internal structure or Inner workings of any “Mathematics or Programming Concept” that matters a lot for a Data Scientist.

This article contains what I’ve learned, and hopefully, it’ll be useful for you as well!

Let’s Kick Start;

Probability and statistics are crucial branches of mathematics that are widely used in various fields such as science, engineering, finance, and economics. These concepts provide the foundation for data analysis, prediction, and decision-making. In this blog, we will explore the basics of probability and statistics using the Julia programming language.

Julia is a high-level, high-performance programming language that is specifically designed for numerical and scientific computing. It provides an extensive library of statistical and graphical techniques, making it an ideal choice for data analysis.

Getting started with Probability in Julia:

Probability is the branch of mathematics that deals with the likelihood of an event occurring. In Julia, we can calculate probabilities using the Distributions package. This package provides a suite of probability distributions and related functions that make it easy to perform probability calculations.

Basics of Probability:

The article demonstrates how to calculate probabilities using the Distributions package in Julia. The article provides examples of how to work with discrete uniform distributions, binomial distributions, and normal distributions. In each example, the article shows how to calculate the mean, probability density function (pdf), and cumulative distribution function (cdf) for a given distribution.

Example:

There are 5 marbles in a bag: 4 are blue, and 1 is red. What is the probability that a blue marble gets picked?

  • Number of ways it can happen: 4 (there are 4 blues)
  • Total number of outcomes: 5 (there are 5 marbles in total)

So the probability = 4/5 = 0.8

We can also show the probability of this certain problem on a Probability Line. Such as;

Probability is always between 0 or 1
  • Discrete uniform distribution:

For example, consider a situation where we have a coin with a probability of heads of 0.5. We can use the Binomial distribution from the Distributions package to calculate the probability of getting exactly 3 heads in 5 coin flips.

using Distributions
d = DiscreteUniform(1, 6) # roll of a dice
mean(d) # 3.5
pdf(d, 1) # 1/6
cdf(d, 3) # 0.5
  • Binomial distribution:
using Distributions
d = Binomial(10, 0.5) # 10 coin flips
mean(d) # 5.0
pdf(d, 5) # 0.246
cdf(d, 5) # 0.62
  • Normal distribution:
using Distributions
d = Normal(0, 1) # standard normal
mean(d) # 0.0
pdf(d, 0) # 0.39
cdf(d, 0) # 0.5

Basics of Statistics:

Statistics is the branch of mathematics that deals with the collection, analysis, interpretation, presentation, and organization of data. In Julia, we can perform various statistical operations using the StatsBase package.

The article demonstrates how to perform various statistical operations using the StatsBase package in Julia. It provides examples of descriptive statistics, hypothesis testing, and linear regression. For example, in the section on descriptive statistics, the article shows how to calculate the mean, median, mode, and variance of a given set of data. In the section on hypothesis testing, the article demonstrates how to perform a two-sample t-test for unequal variances. In the section on linear regression, the article shows how to fit a linear model to a dataset using the GLM package.

Here are some examples:

  • Descriptive statistics:
using StatsBase
x = [1, 2, 3, 4, 5]
mean(x) # 3.0
median(x) # 3.0
mode(x) # 1.0
variance(x) # 2.5
  • Hypothesis testing:
using StatsBase
x = [1, 2, 3, 4, 5]
y = [2, 3, 4, 5, 6]
t_test(x, y) # Two-Sample T-Test (unequal variance)
  • Linear regression:
using GLM, RDatasets
data = dataset("datasets", "mtcars")
fit = lm(@formula(mpg ~ wt), data)
coef(fit) # -5.344

Julia is a great language for probability and statistics, with a wide range of packages available for various tasks. Whether you’re calculating probabilities, performing statistical tests, or fitting models to data, Julia makes it easy to get started and provides high performance for large data sets.

Expert Opinion:

The article provides an overview of using the Julia programming language for Probability and Statistics. It begins by introducing Julia as a high-performance language for numerical and scientific computing, which combines the ease of use of high-level languages such as Python with the performance of low-level languages like C. This makes it an ideal choice for working with probability and statistics.

The article concludes by stating that Julia is a great language for probability and statistics, with a wide range of packages available for various tasks. Whether you are calculating probabilities, performing statistical tests, or fitting models to data, Julia provides a high level of ease and performance.

Probability and Statistics from Scratch in R

Motivation: As part of my personal journey to gain a better understanding of Probability & Statistics, I just follow courses from the world’s best educational Systems such as i.e. MIT, and Stanford. I’ve decided to draw a complete picture of these concepts in R Language. Every time, It is basically, the Internal structure or Inner workings of any “Mathematics or Programming Concept” that matters a lot for a Data Scientist.

This article contains what I’ve learned, and hopefully, it’ll be useful for you as well!

Let’s Kick Start;

What’s the probability?

Aham!

Probability theory is nothing but common sense reduced to calculation.

~Pierre-Simon Laplace

I must start with the relatively simplest example here; When a coin is tossed, there are two possible outcomes:

  • heads (H) or
  • tails (T)

So, the total number of possible outcomes (common sense) is 2 (head or tail). In a mathematical way, we say that the

  • The Probability of coin landing H is 1/2 &
  • The Probability of coin landing T is 1/2
Introduction to Probability and Statistics with R:

Probability and statistics are essential branches of mathematics that are widely used in various fields such as science, engineering, finance, and economics. These concepts provide a foundation for data analysis, prediction, and decision-making. In this blog, we will explore the basics of probability and statistics using the R programming language.

R is a widely used open-source programming language that provides an extensive library of statistical and graphical techniques. It has become an indispensable tool for data analysis and is widely used by statisticians, data scientists, and researchers.

In R-Programming, We have the following distribution to find probabilities such as;

  1. rnorm — draw random numbers from the distribution
  2. dnorm — the density at a given point in the distribution
  3. qnorm — quantile function at a given probability
  4. pnorm — distribution function at input (x)

Example:

Example:

There are 5 marbles in a bag: 4 are blue, and 1 is red. What is the probability that a blue marble gets picked?

  • Number of ways it can happen: 4 (there are 4 blues)
  • Total number of outcomes: 5 (there are 5 marbles in total)

So the probability = 4/5 = 0.8

We can also show the probability of this certain problem on a Probability Line. Such as;

Probability is always between 0 or 1

Getting started with Probability in R:

Probability is the branch of mathematics that deals with the likelihood of an event occurring. In R, we can calculate probabilities using the dbinom() function. This function calculates the probability of observing a specific number of successes in a fixed number of trials, given a success rate.

For example, consider a situation where we have a coin with a probability of heads of 0.5. We can use the dbinom() function to calculate the probability of getting exactly 3 heads in 5 coin flips.

dbinom(3, size = 5, prob = 0.5)

The output of this code will be 0.3125, which is the probability of getting exactly 3 heads in 5 coin flips.

Another commonly used probability distribution in R is the normal distribution. The normal distribution is a continuous probability distribution that is commonly used to model the distribution of a large number of variables. We can use the dnorm() function in R to calculate the probability density of a normal distribution.

For example, consider a normal distribution with a mean of 0 and a standard deviation of 1. We can use the dnorm() function to calculate the probability density at a specific point.

dnorm(0, mean = 0, sd = 1)

The output of this code will be 0.3989422804014327, which is the probability density of the normal distribution at 0.

Getting started with Statistics in R:

Statistics is the branch of mathematics that deals with the collection, analysis, interpretation, presentation, and organization of data. In R, we can perform various statistical tests and analyses using the built-in functions and packages.

One commonly used statistical test in R is the t-test. The t-test is used to compare the means of two groups and determine if they are significantly different from each other. In R, we can perform a t-test using the t.test() function.

For example, consider a situation where we have two groups of data, group A and group B. We can use the t.test() function to perform a t-test to determine if the means of these two groups are significantly different.

t.test(group_A, group_B)

The output of this code will provide the t-statistic, p-value, and other information about the t-test.

Another commonly used statistical analysis in R is linear regression. Linear regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables. In R, we can perform linear regression using the lm() function.

For example, consider a situation where we have data on the number of hours studied and the final exam score. We can use the lm() function to perform linear regression to model the relationship between the number of hours studied and the final exam score.

lm(final_exam_score ~ hours_studied, data = exam_data

Expert Opinion:

In conclusion, R is a powerful tool for performing probability and statistical analysis. It provides a wide range of functions and packages that make it easy for anyone to get started with probability and statistics. Whether you’re a beginner or an experienced statistician, R has everything you need to perform complex analysis and get insights from your data.

In this blog, we have covered the basics of probability and statistics using R. From calculating probabilities using dbinom() and dnorm() to performing t-tests and linear regression using t.test() and lm() functions, respectively. This is just the tip of the iceberg, and there is a lot more to learn and explore in R.

If you’re interested in learning more about probability and statistics with R, I highly recommend checking out online resources such as tutorials, online courses, and books. The R community is vast, and there are many resources available to help you get started.

With the knowledge you’ve gained from this blog, you can now start exploring the world of probability and statistics with R!

Mastering Coturn Server: From Firewall Evasion to WebRTC Deployment and REST API Development:

In this comprehensive guide, I dive into the world of Coturn Server and its capabilities. From understanding its ability to evade firewalls in “Unpacking the Capabilities of Coturn Server: An Analysis of Firewall Evasion”, to scaling and unlocking the potential of WebRTC in the enterprise with “Scaling & Unlocking the Potential of WebRTC in the Enterprise with CoTURN Server Deployment”, and finally exploring the development and deployment of the CoTURN REST API in “CoTURN REST API Development & Deployment”, this guide covers all you need to know about mastering the Coturn Server.

This article introduces a guide that covers the various aspects of Coturn Server. The guide aims to provide a comprehensive understanding of the server, starting with its ability to evade firewalls.

For example, if a user wants to access a website that is blocked by their corporate firewall, the Coturn Server can be used to establish a secure connection to the website and bypass the firewall restrictions. This article provides a detailed understanding of the server’s firewall evasion capabilities.

  • The second article, “Scaling & Unlocking the Potential of WebRTC in the Enterprise with CoTURN Server Deployment”, explains how the Coturn Server can be deployed in an enterprise setting to unlock the full potential of WebRTC. WebRTC (Web Real-Time Communication) is a technology that enables real-time communication over the internet, such as video and voice calls. The article explains how the Coturn Server can be used to scale and optimize WebRTC communication in an enterprise setting.
  • The final article, “CoTURN REST API Development & Deployment”, focuses on the development and deployment of the CoTURN REST API. REST (Representational State Transfer) APIs are a popular way to interact with web-based applications, and the article explains how the CoTURN REST API can be developed and deployed to enhance the functionality of the Coturn Server.

Overall, the guide provides a comprehensive understanding of the Coturn Server, from its firewall evasion capabilities to its role in WebRTC deployment and REST API development.

By following this guide, you will gain a thorough understanding of the capabilities and potential of the Coturn Server, and we are confident that you will find this series informative and enjoyable.

Probability and Statistics from scratch in Python:

Motivation: As part of my personal journey to gain a better understanding of Probability & Statistics, I just follow courses from the world’s best educational Systems such as i.e. MIT, and Stanford. I’ve decided to draw a complete picture of these concepts in Python. Every time, It is basically, the Internal structure or Inner workings of any “Mathematics or Programming Concept” that matters a lot for a Data Scientist.

This article contains what I’ve learned, and hopefully, it’ll be useful for you as well!

Let’s Kick Start;

Probability and statistics are crucial fields in data science and machine learning. They help us understand and make predictions about the behavior of large data sets. In this blog, we will learn about probability and statistics from scratch using Python.

Probabilities

Understanding Probability:

Probability is the study of random events. It measures the likelihood of an event occurring in a particular experiment. There are two main types of probability: classical probability and empirical probability.

Example:

There are 5 marbles in a bag: 4 are blue, and 1 is red. What is the probability that a blue marble gets picked?

  • Number of ways it can happen: 4 (there are 4 blues)
  • Total number of outcomes: 5 (there are 5 marbles in total)

So the probability = 4/5 = 0.8

We can also show the probability of this certain problem on a Probability Line. Such as;

Probability is always between 0 or 1

Classical probability is calculated using the formula:

P(A) = Number of favorable outcomes / Total number of outcomes

Empirical probability is calculated by performing an experiment and counting the number of times an event occurs.

Calculating Probabilities in Python:

To calculate probabilities in Python, we can use the module math. Here’s an example of calculating the probability of rolling a dice and getting a 4:

import math

total_outcomes = 6
favorable_outcomes = 1

probability = favorable_outcomes / total_outcomes
print(probability)

Understanding Statistics:

Statistics

Statistics is the branch of mathematics that deals with the collection, analysis, interpretation, presentation, and organization of data. The primary goal of statistics is to gain insights and make informed decisions based on data.

Descriptive Statistics:

Descriptive statistics deals with the representation of data in a meaningful and concise manner. The most common techniques used in descriptive statistics are measures of central tendency and measures of variability.

Measures of central tendency include mean median and mode. Mean is the average of all the data points, the median is the middle value of the data set, and mode is the most frequently occurring value.

Measures of variability include range, variance, and standard deviation. The range is the difference between the highest and lowest value in a data set, variance is the average of the squared differences from the mean, and the standard deviation is the square root of variance.

Calculating Descriptive Statistics in Python:

To calculate descriptive statistics in Python, we can use the pandas library. Here’s an example of calculating the mean, median, and standard deviation of a data set:

import pandas as pd

data = [1, 2, 3, 4, 5]

mean = pd.Series(data).mean()
median = pd.Series(data).median()
standard_deviation = pd.Series(data).std()

print(f”Mean: {mean}“)
print(f”Median: {median}“)
print(f”Standard Deviation: {standard_deviation}“)

Inferential Statistics:

Inferential statistics deals with making predictions about a population based on a sample of the population. The most common techniques used in inferential statistics are hypothesis testing and regression analysis.

Hypothesis testing is used to determine whether a statistical significance exists between two data sets. Regression analysis is used to predict a dependent variable based on one or more independent variables.

Calculating Inferential Statistics in Python:

To calculate inferential statistics in Python, we can use the scipy library. Here’s an example of performing a t-test:

import scipy.stats as stats

data1 = [1, 2, 3, 4, 5]
data2 = [5, 4, 3, 2, 1]

t_test = stats.ttest_ind
print(f”T-test result: {t_test}“)

Expert Opinion:

Probability and statistics play crucial roles in data science and machine learning. Understanding probability helps us make predictions about random events. Descriptive statistics deals with the representation of data in a meaningful and concise manner, while inferential statistics deals with making predictions about a population based on a sample. With the help of the Python libraries math, pandas, and scipy, we can easily calculate probabilities, descriptive statistics, and inferential statistics.

In this article, I’ve covered the basics of probability and statistics, and how to implement these concepts in Python. I hope you found this article helpful and that it has given you a good starting point for exploring the world of probability and statistics.

Unpacking the Capabilities of Coturn Server: An Analysis of Firewall Evasion

For a comprehensive understanding of CoTURN Server research and development, I invite you to delve into this guide;

Mastering Coturn Server: From Firewall Evasion to WebRTC Deployment and REST API Development

The widespread use of firewalls to secure networks and protect sensitive information has led to the development of various tools and techniques for bypassing them. One such tool is the Coturn Server, which has gained popularity for its ability to penetrate firewalls. In this article, we will unpack the capabilities of the Coturn Server and examine its efficiency in bypassing firewalls.

The Coturn Server is an open-source TURN and STUN server that enables clients behind firewalls to access IP-based multimedia communications. It allows for the bypassing of firewalls by forwarding network traffic from the client to the server and vice versa, making it appear as though the communication is originating from the server.

However, it is important to note that while the Coturn Server may be efficient in bypassing firewalls, its use should be carefully considered. The tool was designed for legitimate purposes, such as providing access to multimedia communications for clients behind firewalls, and its use for malicious purposes is strictly prohibited.

As I mentioned, CoTURN is amongst the most open-source products which have gained huge popularity and recommendations due to their state-of-the-art development on GitHub-like platforms. Let’s analyze the underlying technology and how it is interacting with firewalls.

Difference Between STUN & TURN Protocols:

STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) are both protocols that are used to assist in traversing Network Address Translation (NAT) devices in order to establish peer-to-peer connections for real-time communication applications such as voice and video conferencing.

The main difference between the two is that STUN is used to discover the public IP address of a device behind a NAT, while TURN is used as a relay server when a direct connection between two devices behind NATs is not possible.

In simpler terms, STUN is used to find out the public IP address of a device and establish a direct connection between two devices behind NATs. And TURN is used as a fallback option when direct connection is not possible, it acts as a middleman and forward the data between two devices.

So, technically speaking reliability of WebRTC communication is only dependent on TURN Protocol.

Let’s get a basic understanding of TURN;

Traversal Using Relays around NAT (TURN) [RFC5766] is a protocol that is often used to improve the connectivity of Peer-to-Peer (P2P) applications (as defined in Section 2.7 of [RFC5128]).

TURN allows a connection to be established when one or both sides are incapable of a direct P2P connection. The TURN server is also a building block to support interactive, real-time communication using audio, video, collaboration, games, etc., between two peer web browsers using the Web Real-Time Communication (WebRTC) framework.

The Web Real-Time communication (WebRTC) framework [I-D.ietf-rtcweb-overview] provides protocol building blocks to support direct, interactive, real-time communication using audio, video, collaboration, games, etc., between peers. For peer-to-peer communication, WebRTC uses Interactive Connectivity Establishment (ICE) protocol [RFC5245] for gathering address candidates and doing connectivity checks.

Enterprise-Level Firewall policy:

It is quite important to understand how enterprises establish their firewalls. Please spend some time on the below paragraph to digest the salt;

Enterprise typically has a white-list of permitted outside applications/sites and can blacklist HTTP(S) connections via various forms of detections (DNS lookup, ALPN, HTTP URL Filtering, DPI proxy that at least performs HTTPS inspection of SNI in TLS exchange and validates SSL records, HTTP(S) proxy etc.).

A firewall in this configuration would block TCP/UDP connections with external peers on the Internet and arbitrary TURN servers. For example, a firewall would block the usage of STUN with external peers and force the clients to use an enterprise-provided TURN server for all external WebRTC media communications.

Why is UDP simply blocked in enterprises?

Because most protocols use TCP, many enterprise firewalls are configured to simply block UDP.

Security policies of enterprise networks typically require filtering incoming unsolicited traffic, blocking certain protocols, and doing application-level filtering and scanning for spam, malware, and intellectual property. Most of these protocols run over TCP and are well supported by firewalls and application-specific proxy functions.

Enterprise Level Firewall Traversal using TURN Server:

Enterprise firewalls are configured to permit UDP, and TCP traffic to well-known TURN servers to allow WebRTC media streams and data channels.

Enterprise TURN server can use third-party authorization http://tools.ietf.org/html/draft-ietf-tram-turn-third-party-authz-04 to distinguish between business and social calls.

The enterprise-provided TURN server will have to support both first and third-party authorization. This way client can use third-party authorization for WebRTC servers that have business relationships with the Enterprise network and use first-party authentication for non-business related calls.

Executing ICE Candidate Tests using a Bash Script:

In this section, we will be discussing a bash script that has been created to perform the task of trickling ICE candidates and test the arguments regarding coturn. The script leverages the trickle-ice tool to perform this task, which is an essential component for implementing real-time communication applications such as voice and video conferencing. The script allows us to test the reliability of the WebRTC communication, which is dependent on the TURN protocol. Here’s a basic bash script to trickle ICE candidates and test the arguments discussed in the above description;

#!/bin/bash

# Define the STUN and TURN servers
STUN_SERVER=“stun.l.google.com:19302”
TURN_SERVER=“turn:your-turn-server.com:3478”

# Start trickle ICE candidates
./trickle-ice -u $STUN_SERVER -t $TURN_SERVER

# Check if direct connection is possible with STUN
if [ $? -eq 0 ]; then
echo “Direct connection established with STUN.”
else
echo “Direct connection not possible. Using TURN as fallback.”
fi

# Check enterprise firewall policy
if [ -n “$(grep “Enterprise firewall blocks UDP” /etc/firewall.conf)“ ]; then
echo “Enterprise firewall blocks UDP. Using TURN for WebRTC media communication.”
else
echo “Enterprise firewall allows UDP. Using STUN for WebRTC media communication.”
fi

# Check if TURN server supports third-party authorization
if [ -n “$(grep “TURN server supports third-party auth” /etc/turn.conf)“ ]; then
echo “TURN server supports third-party authorization. Using third-party auth for business calls.”
else
echo “TURN server does not support third-party authorization. Using first-party auth for all calls.”
fi

This script assumes the availability of the trickle-ice tool and the existence of the /etc/firewall.conf and /etc/turn.conf configuration files. The script also assumes that the TURN server address and port are specified in the TURN_SERVER variable.

The script can be adjusted or extended based on the specific requirements of your setup.

To install the trickle-ice tool and run the script, follow these steps:

  • Install trickle-ice using the following command:
sudo apt-get install trickle-ice
  • Create a new file with .shextension and paste the script content inside.
  • Make the file executable with the following command:
chmod +x validate_ice_servers.sh
  • Run the script with the following command:
./validate_ice_servers.sh

The code is a shell script that tests various components required for WebRTC media communication. It starts by defining the STUN and TURN servers used for this communication and runs the “trickle-ice” command to check if a direct connection is possible with the STUN server. If a direct connection is not possible, it uses TURN as a fallback.

Next, it checks the enterprise firewall policy by searching for a specific string in the “/etc/firewall.conf” file. If the string is found, it means that the enterprise firewall blocks UDP traffic, and TURN must be used for WebRTC media communication. If the string is not found, it means that the firewall allows UDP traffic, and STUN can be used for this communication.

Finally, the code checks if the TURN server supports third-party authorization by searching for a specific string in the “/etc/turn.conf” file. If the string is found, it means that the TURN server supports third-party authorization, and this will be used for business calls. If the string is not found, the TURN server does not support third-party authorization, and first-party authorization will be used for all calls.

Expert Opinion:

Enterprise firewalls are security systems that are used to protect a company’s network and data. Despite the fact that these firewalls typically block the transmission of data using the User Datagram Protocol (UDP), they make an exception for the transmission of UDP and Transmission Control Protocol (TCP) traffic to well-known TURN (Traversal Using Relays around NAT) servers.

This allows for the use of WebRTC (Web Real-Time Communication) technology, which enables real-time media streaming and data transfer between web browsers. To ensure secure and controlled communication, the TURN server software known as Coturn implements both first-party and third-party authorization. This differentiation between communication for business purposes and social communication helps to ensure that the correct level of security is maintained for each type of communication.