Need for Packet Truncation

Reduce Storage: Network traffic payload can be very large (64 to 9216), and storing everything will be expensive. Packet truncation helps reduce the amount of data that needs to be stored by removing irrelevant or redundant information from packets.

Reduce CPU Cycles: Truncated packets require less processing to analyze, which can improve the overall speed and performance of the tools.

Simplify Analysis: Network administrators can easily identify network performance issues more quickly and efficiently since truncated packets have the relevant portions of the packet.

Improve Security: By removing sensitive information from the payload, security can be improved by limiting the exposure of confidential data.

Open Packet Broker for Truncation

Aviz Open Packet Broker industry first Packet broker solution built on Open Networking NOS SONiC supporting truncation on commodity ASICs supporting wire-speed packet truncation. Open Packet broker truncation has the following capabilities.

• Packet Truncation based on custom offsets (48 bytes to 4094 bytes).
• VLAN tag insertion for truncated packets for different tooling purposes.
• Load Balance across tools for optimal processing 

Packet Truncation or slicing will allow only the user-defined byte from an incoming packet, and the remaining bytes are discarded. This helps in reducing the quantity of data processed on the tool port.

Conclusion

Packet Truncation helps reduce storage requirements, improve analysis, speed up processing, and enhance network security. Open Packet Broker from Aviz Aviz OPB  improves cost savings by providing customers the choice of Open Networking HW SKUs supporting line-rate packet processing.

Authors: Chid Perumal, CTO, and Rajasekaran S, Member of Technical Staff, Aviz Networks

The post Slice and Save Costs with Open Packet Broker appeared first on Linux.com.

GTP (GPRS Tunneling) is a group of IP-based communications protocols used to carry GPRS traffic within mobile GSM networks. It works as a carrier for mobile packets over an underlay IP network using tunneling. GTP is used between the base station and the gateway, which are part of the mobile elements in 5G transport architecture. The packet is encapsulated over IP and delivered across the IP network.

Why do we need GTP Parsing and Filtering?

Network monitoring tools require inner header information for the mobile network for threat monitoring, analysis, and inspection. So, network packet brokers (NPB) residing in the GPRS core networks need to filter, forward, and load balance packets toward the tools for inspection. This requires NPBs to have the capability to filter based on outer and inner headers to identify the GTP sessions in the data stream to control data flow within your infrastructure. This deep packet inspection will result in the decision-making of allowing or denying traffic based on the packet policies from the mobile operator station.

A major challenge in today’s mobile network is the data traffic from the user equipment, and its application is rapidly growing. To effectively monitor the performance and obtain a better quality of service, service providers should be able to correlate the traffic flow based on each subscriber’s data and service gateway tunnel endpoint identifiers (TEID). Therefore, GTP user and control packets need to be parsed by NPBs in the core GPRS network and packets towards the underlay IP. 

Open Networking Approach 

The evolution of modern ASICs in their programmability, providing flexible parsers for filtering, and TCAM-scale, has created an opportunity for using them on Network Packet Brokers for the 5G mobile network to perform deep packet inspection of GTP sessions. SONiC open-source NOS, regarded as the “Linux of Networking,” supports these modern ASICs. The flexible micro-services-based software architecture exposing the ASIC capabilities using standardized SAI (Switch Abstraction Interface) has created a clear opportunity to build network packet brokers for 5G deployments.

Aviz’s Open Packet Broker (OPB) is the industry’s first software-based microservice built on SONiC using ASIC (NVIDIA Spectrum) programmability capabilities to provide deep insights on 5G mobile traffic.

flow flow1
network-ports Ethernet13/1
tool-ports Ethernet16/1
tool-ports port-channel1
rule 1 permit src-ip 1.1.1.1/32 dest-ip 2.2.2.2/32 protocol tcp gtp "teid 0x13467254 inner-sip 3.3.3.3/32 inner-dip 4.4.4.4/32 inner-protocol udp inner_l4srcport 567 inner_l4destport 789" counters enable
rule 2 permit src-ip 2401::1 src-netmask f::f dest-ip 2401::2 dest-netmask f::f protocol udp l4portsrc 789 l4portdst 456 gtp "teid 0x11112222 inner-sip 1203::1 inner-smask f::f inner-dip 1203::2 inner-dmask f::f inner-protocol tcp inner_l4srcport 909 inner_l4destport 657" counters enable

Figure 1: Simple (IPv4/IPv6) Rule configuration for GTP session monitoring with LoadBalancing

Figure 2: GTP configuration using APIs

Conclusion

By providing 5G’s high capacity, low latency, and massive connectivity to customers, mobile carriers must ensure uninterrupted network service with a higher quality of experience. Therefore, mobile operators need a cost-effective solution that can meet the increase in speeds and provide deep inspection. Aviz leverages the strengths of the open networking ecosystem for both hardware and software to provide mobile network operators with the solution that’s key to greater QoE at a lower cost: OPB (Open Packet Broker).

Authors: Chid Perumal, CTO, and Rajasekaran S, Member of Technical Staff, Aviz Networks

The post Increasing 5G Quality of Experience (QoE) Using SONiC and Open Packet Broker appeared first on Linux.com.

The post Open Source Networks in Action: How leading telcos are harnessing the power of LF Networking appeared first on Linux Foundation.

The post Open Source Networks in Action: How leading telcos are harnessing the power of LF Networking appeared first on Linux.com.

SAN FRANCISCO– April 14, 2021 –  Today, the Linux Foundation, the nonprofit organization enabling mass innovation through open source, announced the Software for Open Networking in the Cloud (SONiC, an open source networking operating system), is now part of the Linux Foundation. The Linux Foundation provides a venue for continued ecosystem, developer growth and diversity, as well as collaboration across the open source networking stack.  

“We are pleased to welcome SONiC to the Linux Foundation family of open networking projects,” said Arpit Joshipura, general manager, Networking, Edge, and IoT, the Linux Foundation. “SONiC is a  leader in open source data center NOS deployments, and we’re looking forward to growing its developer community.” 

The Linux Foundation will primarily focus on the software component of SONiC, and continue to partner with Open Compute Platform(OCP) on aligning hardware and specifications like SAI. 

“Microsoft founded SONiC to bring high reliability and fast innovation to the routers in Azure cloud data centers. We created it as open source so the entire networking ecosystem would grow stronger.  SONiC already runs on millions of ports in the networks of cloud scalers, enterprises, and fintechs.  The SONiC project is thrilled to be joining the Linux Foundation to take the community to its next jump in scale, participation, and usage,” said  Dave Maltz, Technical Fellow and Corporate Vice President, Microsoft Azure Networking.

About SONiC

Created by Microsoft for its Azure data centers, SONiC is an open source network operating system (NOS)  based on Linux that runs on over 100 different switches from multiple vendors and ASICs. It offers a full-suite of network functionality, like BGP and RDMA, that has been production-hardened in the data centers of some of the largest cloud-service providers. It offers teams the flexibility to create the network solutions they need while leveraging the collective strength of a large ecosystem and community. 

Existing Ecosystem

SONiC brings a strong existing ecosystem, with premier members including Alibaba, Broadcom, Dell, Google, Intel, Microsoft, NVIDIA and 50+ global partners. The SONiC community will host its first hackathon later this year. Stay tuned for details and registration information. More information about SONiC, including how to join, is available at SONiC (azure.github.io).

Support from Key Stakeholders & Customers

Alibaba

“This is a big milestone for the SONiC community. After joining the Linux Foundation, the SONiC community will play a much more important role in the networking ecosystem,” said Dennis Cai, Head of Network Infrastructure, Alibaba Cloud. “Congratulations!  As one of the pioneering SONiC users and contributors, Alibaba Cloud has widely deployed SONiC- based whitebox switches in our data centers, edge computing cloud, P4- based network gateways, and will extend the deployment to Wide Area Networks. With modern network OS design and operation- friendly features, we already gained tremendous value from the large-scale deployments. Alibaba is committed to the SONiC community, and will continue bringing our large-scale deployment best practices to the community, such as open hardware specs , network in-band telemetry, high performance networking, and network resiliency features, SRv6, etc.” 

Broadcom

“Large hyperscalers agree that merchant silicon, hardware independence, and open source protocol and management stack are essential for running their data center networks. Broadcom has wholeheartedly supported this vision with leading-edge, predictable silicon execution and contributions to the SONiC project. We are excited to see the SONiC initiative join the Linux Foundation and look forward to working with the streamlined ecosystem to drive the data center and hyperscale needs of the future,” said Mohammad Hanif, senior director of engineering, Core Switching Group, Broadcom.

Dell Technologies 

“We believe SONiC will continue its accelerated adoption into the modern data center, delivering the scale, flexibility and programmability needed to run enterprise-level networks,” said Dave Lincoln, vice president of product management at Dell Technologies. “As a leading SONiC contributor, we see the advantages it brings to the supporting open source community and customers. As we continue the drive to take open-source-based solutions mainstream, we look forward to working with the Linux Foundation and its supporting communities to drive SONIC’s development and adoption.”

EBay

“eBay operates a large-scale network infrastructure to support its growing global business. eBay cares about the openness and quality of NOS to operate its network infrastructure. eBay is an active participant in the SONiC community and deploys SONiC at scale in its infrastructure. eBay is excited to see this next step of growth of the SONiC community,” said Parantap Lahiri, vice president, Network and Datacenter Engineering at eBay. 

EPFL

“At EPFL, we have been looking for a vendor neutral and flexible NOS that can provide HaaS capabilities for our Private Cloud Environment. SONiC OS provides us the solution we have been looking for in our Data Centre, allowing us to migrate to a powerful and modern Data Centre network. We are looking forward to this next phase in the SONiC community,” said Julien Demierre, Network and System architect at EPFL.

Google

“We believe moving SONiC to the Linux Foundation is very important as it will further enhance collaboration across the open source network, community and ecosystem. Google has more than a decade of experience in SDN; our data centers and WAN are exclusively SDN controlled, and we are excited to have helped bring SDN capabilities to SONiC . We fully support the move to the LF and intend to continue making significant upstream contributions to drive feature velocity and make it easier for operators to realize the benefits of SDN with PINS/SONiC and P4,” said  Dan Lenoski, vice president, Engineering, Network Infrastructure, Google. 

Intel 

“Intel has a strong history of working with SONiC and the Linux Foundation to help to propel innovation in an open, cooperative environment where ideas are shared and iterated.  We continually promote open collaboration, encompassing open-source technologies such as the Infrastructure Programmer Developer Kit and P4 integrated networking stack (PINS), using Intel Xeon Scalable processors, Infrastructure Processing Units and Tofino Intelligent Fabric Processors as base hardware,” said Ed Doe, vice president and general manager, Switch and Fabric Group at Intel. “Joining the Linux Foundation will help SONiC to flourish, and in turn create greater benefit for cloud service providers, network operators and enterprises to create customized network solutions and transform data-intensive workloads from data center to the edge.”

NVIDIA

“This is an important milestone for SONiC and the community behind it,” said Amit Katz, vice president of Ethernet Switches at NVIDIA. “NVIDIA is committed to supporting the community version of SONiC that is 100 percent open source, enabling data center operators to control the code inside their cloud fabrics, accelerated by state-of-the-art platforms with SONiC support, such as NVIDIA’s Spectrum family of switches.” 

Open Compute Project 

“The Open Compute Project Foundation is pleased to continue its collaboration with SONIC as part of the OCP’s new hardware – software co-design strategy. The open source SONiC Network Operating System is enabling rapid innovation across the network ecosystem, and it began with the definition of the Switch Abstraction Interface (SAI) at OCP.   Hardware – software co-design focuses on software that requires intimate knowledge of the hardware to drive maximum hardware performance, and speed time-to-market for hardware where system performance and ecological footprint can be highly dependent on software and hardware interactions,” said George Tchaparian, CEO Open Compute Project Foundation.

About the Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and commercial adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

# # #

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

The post Software for Open Networking in the Cloud (SONiC) Moves to the Linux Foundation appeared first on Linux Foundation.

The post Software for Open Networking in the Cloud (SONiC) Moves to the Linux Foundation appeared first on Linux.com.

SAN FRANCISCO, April 14, 2022 – Today, the Zephyr® Project announced that T-Mobile has joined as a Platinum member, leveraging the Real-Time Operating System (RTOS) to power its new Developer Kit, which gives innovators fast and easy access to build on T-Mobile’s network. The Zephyr Project is an open source project at the Linux Foundation that builds a safe, secure and flexible RTOS for resource-constrained devices. T-Mobile is the first wireless carrier to join the project.

“As a leader in the industry and our first telecom member, T-Mobile brings a unique perspective and expertise to the Zephyr ecosystem,” said Kate Stewart, Vice President of Dependable Embedded Systems at The Linux Foundation. “Zephyr’s existing wireless capabilities (Bluetooth Low Energy, Wi-Fi, and 802.15.4), coupled with DevEdge, T-Mobile’s new developer platform, will unleash innovators to create new solutions for the connected future.”

Zephyr RTOS is easy to deploy, secure, connect and manage and supports more than 350 boards running embedded microcontrollers from Arm and RISC-V to Tensilica, NIOS, and ARC as single and multicore systems. It has a growing set of software libraries that can be used across various applications and industry sectors such as Industrial IoT, wearables, machine learning and more. Zephyr is built with an emphasis on broad chipset support, security, dependability, long-term support releases and a growing open source ecosystem.

“T-Mobile is thrilled to be the first wireless provider to join the Zephyr Project. As we shared when we launched DevEdge earlier this month, we envision a future where everything that can be connected, will be. And that requires massive innovation.” said Rob Roy, SVP of Emerging Business Innovation at T-Mobile. “Zephyr’s RTOS will help T-Mobile enable developers to build better and faster, unlocking massive innovation on our network.”

T-Mobile’s new Developer Kit, which will run on Zephyr RTOS, gives developers immediate access to T-Mobile’s network – no out-of-pocket costs, no testing hardware, no lengthy build time required. And for a limited time, T-Mobile is giving away Developer Kits for free while supplies last to developers who sign up now. To learn more, and to sign-up for a kit, developers can visit devedge.t-mobile.com/solutions/iot-developer-kit.

T-Mobile joins other Platinum members including Antmicro, Baumer, Google, Intel, Meta, Nordic Semiconductor, NXP, Oticon and Qualcomm Innovation Center. T-Mobile will join the Zephyr Governing Board and its commitment to ensure balanced collaboration and feedback that meets the needs of its community.

Other Zephyr Project members include AVSystem, BayLibre, Beijing University of Posts and Telecommunications (BUPT), Eclipse Foundation, FIWARE, Foundries.io, Golioth, Infineon, Institute of Communication and Computer Systems (ICCS), Laird Connectivity, Linaro, Memfault, Northeastern University, Parasoft, Percepio, Research Institute of Sweden (RISE), RISC-V, SiFive, Silicon Labs, Synopsys, Texas Instruments and Wind River.

Zephyr Developer Summit

The Zephyr community will gather virtually and in-person at the Computer History Museum in Mountain View, California, on June 8-9. The second annual Zephyr Developer Summit will feature speakers from Antmicro, AVSystem, Bitergia, Boston Technology Law, Entropic Engineering, Circuit Dojo, Facebook/Meta, Golioth, Google, Huawei, Intel, Laird Connectivity, Lattix, Linaro, The Linux Foundation, Nordic Semiconductor, Percepio, Samsung, ST Microelectronics, Synopsys, Wind River and Zonneplan.

The Summit is open to the public with various registration rates to attend in-person or virtually. Learn more and register here: https://events.linuxfoundation.org/zephyr-developer-summit/register/.

A few of highlights of the Zephyr Developer Summit include:

An Intro to Zephyr Day on June 7 that offer several presentations and overviews for new developers. It will also feature in-depth hands-on tutorials from Golioth and Nordic Semiconductor.A Mini-Conference for Testing & Traceability that features sessions about design and testing, unit tests and emulators, new framework for testing fleet of platforms, and a Birds of a Feather (BoF) for quality and testing processes for Zephyr.A Mini-Conference for RISC-V collaboration with presentations about SMP support, what it is currently and what lies ahead, as well as the use of the RISC-V architecture in the Zephyr ecosystem.

The complete schedule for the Summit can be found here. The Zephyr Developer Summit is made possible thanks to Diamond Sponsors Antmicro, Google and Intel; Platinum Sponsor Nordic Semiconductor; Gold Sponsor NXP; Silver Sponsors Golioth and Memfault and Session Recording Sponsor BayLibre.

Last year, almost 700 people registered for the first-ever virtual Zephyr Developer Summit in June. The event consisted of 5 mini-conferences, 28 sessions and 51 speakers who presented technical content, best practices, real-world use cases and more. Videos are available on the Zephyr Project YouTube Channel.

To learn more about Zephyr RTOS, visit the Zephyr website and blog.

About the Zephyr Project

The Zephyr Project is an open source, scalable real-time operating system (RTOS) supporting multiple hardware architectures. To learn more, please visit www.zephyrproject.org.

About the Linux Foundation

Founded in 2000, the Linux Foundation is supported by more than 1,000 members and is the world’s leading home for collaboration on open source software, open standards, open data, and open hardware. Linux Foundation’s projects are critical to the world’s infrastructure including Linux, Kubernetes, Node.js, and more.  The Linux Foundation’s methodology focuses on leveraging best practices and addressing the needs of contributors, users and solution providers to create sustainable models for open collaboration. For more information, please visit us at linuxfoundation.org.

###

The post T-Mobile joins the Zephyr Project as Platinum Member appeared first on Linux Foundation.

The post T-Mobile joins the Zephyr Project as Platinum Member appeared first on Linux.com.

Linux has been around in various forms for about 30 years, and the kernel is the basis of other operating systems such as Android and Chrome OS. Supercomputers use it at one end of the computing spectrum and in embedded devices at the other. Linux is used on laptops, desktop computers, and servers in between these extremes.

And it’s also used in single-board computers — this category includes popular devices like the Raspberry Pi.

Therefore it’s fair to say that Linux has been widely adopted.

While microcontrollers and lean, mean software frameworks necessarily dominate small electronic products that are generally single-purpose devices and have modest processing requirements, Linux meets the needs of another important subset. Some products have multiple features that need to be available concurrently. Some cases may require significant processor power and need RAM measured in gigabytes rather than the kilobytes of RAM more typically found in microcontrollers. IP security cameras are based on Linux. They can stream live video, respond to motion detection events, identify human faces in video streams in real-time, record video to an SD card, transfer files over FTP, and host a web server for management and configuration purposes. That mix of concurrently available functionality requires both sufficiently powerful hardware and an operating system that supports multiple processes and threads, provides a capable file system, and has a wide selection of applications readily available for it. Linux is a perfect fit. And it’s open source and free.

Bluetooth Technology and Linux

Bluetooth^® technology can be used on Linux. The controller part of the Bluetooth stack is typically a system on a chip that is either an integral part of the mainboard or implemented in a peripheral like a USB dongle. The host part of the Bluetooth stack runs as a system service, and the standard Linux Bluetooth host implementation is called BlueZ.

BlueZ supports both the Bluetooth LE Peripheral and Central roles using GAP and GATT and Bluetooth mesh, provided the underlying controller supports dependent Bluetooth features. And its multi-process architecture means that multiple Bluetooth applications can be running simultaneously on a single device, which offers some exciting possibilities.

But for a developer, working with Bluetooth technology on Linux for the first time can be challenging. BlueZ defines a straightforward, logical API, but the way a developer must use it in applications is dissimilar to how a developer works with Bluetooth APIs on most other platforms. This is a consequence of the system’s architecture, which, whilst not unique, is typically very visible to the developer and usually needs to be well understood so that those logical BlueZ APIs can be used.

The post Understanding Bluetooth Technology for Linux appeared first on Linux.com.

As we wrap up the second year of living through a global pandemic,  I wanted to take a moment to both look ahead to next year, as well as recognize how the open networking and edge industry has shifted over the past year.  Read below for a  list of what we can expect in 2022, as well as a brief “report card” on where my industry predictions from last year landed. 

1. Dis-aggregation will enter the “Re-aggregation” phase (in terms of software, organizations, and industries) 

This will be enabled by Super Blueprints (which bring end-to-end open source projects together), and we’ll see more multi-org collaboration (e.g., Standards Bodies, Alliances, and Foundations) re-aggregating to solve common problems. Edge computing will serve as the glue that binds common IoT frameworks together across vertical industries. 

2. Realists and Visionaries will fight it out for dollars and productivity 

Given that what started as a pandemic could become endemic, there will be an internal tussle between Realists (making money off of 4G), Engineers currently coding 5G,  and Visionaries looking to 6G and beyond. (In other words, the cycle continues). 

3. Security will emerge as the key differentiator in Open source

Collaboration among governments and other global organizations against “bad actors” will penetrate geopolitical walls to bring a global ecosystem together, via open source. 

4. Market Analysts will reinvent themselves

There is no longer a clear way to track Cloud, Telecom, Enterprise, and other markets individually. There is a big market realignment in progress, with new killer use cases. 

5. Seamless Vertical industries will emerge

Enabled by Open Source Software — many vertical industries will not even know (or care) how the pipe traverses across their last mile to central cloud and edges (led by Manufacturing, Retail, Energy, Healthcare & Automotive). 

What did I miss? I would love to have your comments on LinkedIn.

Now let’s take a look at where my predictions from last year actually landed…

See my 2021 predictions from last year: https://www.lfnetworking.org/blog/2020/12/15/predictions-2021-networking-edge/ 

The post Vision 2022: Open Networking & Edge Predictions appeared first on Linux.com.

In 2021, The Linux Foundation continued to see organizations embrace open collaboration and open source principles, accelerating new innovations, approaches, and best practices. As a community, we made significant progress in the areas of cloud-native computing, 5G networking, software supply chain security, 3D gaming, and a host of new industry and social initiatives.

Download and read the report today.

The post Download the 2021 Linux Foundation Annual Report appeared first on Linux.com.

Jason Perlow, Editorial Director at the Linux Foundation interviews Thomas Gleixner, Linux Foundation Fellow, CTO of Linutronix GmbH, and project leader of the PREEMPT_RT real-time kernel patch set.

JP: Greetings, Thomas! It’s great to have you here this morning — although for you, it’s getting late in the afternoon in Germany. So PREEMPT_RT, the real-time patch set for the kernel is a fascinating project because it has some very important use-cases that most people who use Linux-based systems may not be aware of. First of all, can you tell me what “Real-Time” truly means? 

TG: Real-Time in the context of operating systems means that the operating system provides mechanisms to guarantee that the associated real-time task processes an event within a specified period of time. Real-Time is often confused with “really fast.” The late Prof. Doug Niehaus explained it this way: “Real-Time is not as fast as possible; it is as fast as specified.”

The specified time constraint is application-dependent. A control loop for a water treatment plant can have comparatively large time constraints measured in seconds or even minutes, while a robotics control loop has time constraints in the range of microseconds. But for both scenarios missing the deadline at which the computation has to be finished can result in malfunction. For some application scenarios, missing the deadline can have fatal consequences.

In the strict sense of Real-Time, the guarantee which is provided by the operating system must be verifiable, e.g., by mathematical proof of the worst-case execution time. In some application areas, especially those related to functional safety (aerospace, medical, automation, automotive, just to name a few), this is a mandatory requirement. But for other scenarios or scenarios where there is a separate mechanism for providing the safety requirements, the proof of correctness can be more relaxed. But even in the more relaxed case, the malfunction of a real-time system can cause substantial damage, which obviously wants to be avoided.

JP: What is the history behind the project? How did it get started?

TG: Real-Time Linux has a history that goes way beyond the actual PREEMPT_RT project.

Linux became a research vehicle very early on. Real-Time researchers set out to transform Linux into a Real-Time Operating system and followed different approaches with more or less success. Still, none of them seriously attempted a fully integrated and perhaps upstream-able variant. In 2004 various parties started an uncoordinated effort to get some key technologies into the Linux kernel on which they wanted to build proper Real-Time support. None of them was complete, and there was a lack of an overall concept. 

Ingo Molnar, working for RedHat, started to pick up pieces, reshape them and collect them in a patch series to build the grounds for the real-time preemption patch set PREEMPT_RT. At that time, I worked with the late Dr. Doug Niehaus to port a solution we had working based on the 2.4 Linux kernel forward to the 2.6 kernel. Our work was both conflicting and complimentary, so I teamed up with Ingo quickly to get this into a usable shape. Others like Steven Rostedt brought in ideas and experience from other Linux Real-Time research efforts. With a quickly forming loose team of interested developers, we were able to develop a halfway usable Real-Time solution that was fully integrated into the Linux kernel in a short period of time. That was far from a maintainable and production-ready solution. Still, we had laid the groundwork and proven that the concept of making the Linux Kernel real-time capable was feasible. The idea and intent of fully integrating this into the mainline Linux kernel over time were there from the very beginning.

JP: Why is it still a separate project from the Mainline kernel today?

TG: To integrate the real-time patches into the Linux kernel, a lot of preparatory work, restructuring, and consolidation of the mainline codebase had to be done first. While many pieces that emerged from the real-time work found their way into the mainline kernel rather quickly due to their isolation, the more intrusive changes that change the Linux kernel’s fundamental behavior needed (and still need) a lot of polishing and careful integration work. 

Naturally, this has to be coordinated with all the other ongoing efforts to adopt the Linux kernel to the different use cases ranging from tiny embedded systems to supercomputers. 

This also requires carefully designing the integration so it does not get in the way of other interests and imposes roadblocks for further developing the Linux kernel, which is something the community and especially Linus Torvalds, cares about deeply. 

As long as these remaining patches are out of the mainline kernel, this is not a problem because it does not put any burden or restriction on the mainline kernel. The responsibility is on the real-time project, but on the other side, in this context, there is no restriction to take shortcuts that would never be acceptable in the upstream kernel.

The real-time patches are fundamentally different from something like a device driver that sits at some corner of the source tree. A device driver does not cause any larger damage when it goes unmaintained and can be easily removed when it reaches the final state bit-rot. Conversely, the PREEMPT_RT core technology is in the heart of the Linux kernel. Long-term maintainability is key as any problem in that area will affect the Linux user universe as a whole. In contrast, a bit-rotted driver only affects the few people who have a device depending on it.

JP: Traditionally, when I think about RTOS, I think of legacy solutions based on closed systems. Why is it essential we have an open-source alternative to them? 

TG: The RTOS landscape is broad and, in many cases, very specialized. As I mentioned on the question of “what is real-time,” certain application scenarios require a fully validated RTOS, usually according to an application space-specific standard and often regulatory law. Aside from that, many RTOSes are limited to a specific class of CPU devices that fit into the targeted application space. Many of them come with specialized application programming interfaces which require special tooling and expertise.

The Real-Time Linux project never aimed at these narrow and specialized application spaces. It always was meant to be the solution for 99% of the use cases and to be able to fully leverage the flexibility and scalability of the Linux kernel and the broader FOSS ecosystem so that integrated solutions with mixed-criticality workloads can be handled consistently. 

Developing real-time applications on a real-time enabled Linux kernel is not much different from developing non-real-time applications on Linux, except for the careful selection of system interfaces that can be utilized and programming patterns that should be avoided, but that is true for real-time application programming in general independent of the RTOS. 

The important difference is that the tools and concepts are all the same, and integration into and utilizing the larger FOSS ecosystem comes for free.

The downside of PREEMPT_RT is that it can’t be fully validated, which excludes it from specific application spaces, but there are efforts underway, e.g., the LF ELISA project, to fill that gap. The reason behind this is, that large multiprocessor systems have become a commodity, and the need for more complex real-time systems in various application spaces, e.g., assisted / autonomous driving or robotics, requires a more flexible and scalable RTOS approach than what most of the specialized and validated RTOSes can provide. 

That’s a long way down the road. Still, there are solutions out there today which utilize external mechanisms to achieve the safety requirements in some of the application spaces while leveraging the full potential of a real-time enabled Linux kernel along with the broad offerings of the wider FOSS ecosystem.

JP: What are examples of products and systems that use the real-time patch set that people depend on regularly?

TG: It’s all over the place now. Industrial automation, control systems, robotics, medical devices, professional audio, automotive, rockets, and telecommunication, just to name a few prominent areas.

JP: Who are the major participants currently developing systems and toolsets with the real-time Linux kernel patch set?  

TG: Listing them all would be equivalent to reciting the “who’s who” in the industry. On the distribution side, there are offerings from, e.g., RedHat, SUSE, Mentor, and Wind River, which deliver RT to a broad range of customers in different application areas. There are firms like Concurrent, National Instruments, Boston Dynamics, SpaceX, and Tesla, just to name a few on the products side.

RedHat and National Instruments are also members of the LF collaborative Real-Time project.

JP: What are the challenges in developing a real-time subsystem or specialized kernel for Linux? Is it any different than how other projects are run for the kernel?

TG: Not really different; the same rules apply. Patches have to be posted, are reviewed, and discussed. The feedback is then incorporated. The loop starts over until everyone agrees on the solution, and the patches get merged into the relevant subsystem tree and finally end up in the mainline kernel.

But as I explained before, it needs a lot of care and effort and, often enough, a large amount of extra work to restructure existing code first to get a particular piece of the patches integrated. The result is providing the desired functionality but is at the same time not in the way of other interests or, ideally, provides a benefit for everyone.

The technology’s complexity that reaches into a broad range of the core kernel code is obviously challenging, especially combined with the mainline kernel’s rapid change rate. Even larger changes happening at the related core infrastructure level are not impacting ongoing development and integration work too much in areas like drivers or file systems. But any change on the core infrastructure can break a carefully thought-out integration of the real-time parts into that infrastructure and send us back to the drawing board for a while.

JP:  Which companies have been supporting the effort to get the PREEMPT_RT Linux kernel patches upstream? 

TG: For the past five years, it has been supported by the members of the LF real-time Linux project, currently ARM, BMW, CIP, ELISA, Intel, National Instruments, OSADL, RedHat, and Texas Instruments. CIP, ELISA, and OSADL are projects or organizations on their own which have member companies all over the industry. Former supporters include Google, IBM, and NXP.

I personally, my team and the broader Linux real-time community are extremely grateful for the support provided by these members. 

However, as with other key open source projects heavily used in critical infrastructure, funding always was and still is a difficult challenge. Even if the amount of money required to keep such low-level plumbing but essential functionality sustained is comparatively small, these projects struggle with finding enough sponsors and often lack long-term commitment.

The approach to funding these kinds of projects reminds me of the Mikado Game, which is popular in Europe, where the first player who picks up the stick and disturbs the pile often is the one who loses.

That’s puzzling to me, especially as many companies build key products depending on these technologies and seem to take the availability and sustainability for granted up to the point where such a project fails, or people stop working on it due to lack of funding. Such companies should seriously consider supporting the funding of the Real-Time project.

It’s a lot like the Jenga game, where everyone pulls out as many pieces as they can up until the point where it collapses. We cannot keep taking; we have to give back to these communities putting in the hard work for technologies that companies heavily rely on.

I gave up long ago trying to make sense of that, especially when looking at the insane amounts of money thrown at the over-hyped technology of the day. Even if critical for a large part of the industry, low-level infrastructure lacks the buzzword charm that attracts attention and makes headlines — but it still needs support.

JP:  One of the historical concerns was that Real-Time didn’t have a community associated with it; what has changed in the last five years?  

TG: There is a lively user community, and quite a bit of the activity comes from the LF project members. On the development side itself, we are slowly gaining more people who understand the intricacies of PREEMPT_RT and also people who look at it from other angles, e.g., analysis and instrumentation. Some fields could be improved, like documentation, but there is always something that can be improved.

JP:  What will the Real-Time Stable team be doing once the patches are accepted upstream?

TG: The stable team is currently overseeing the RT variants of the supported mainline stable versions. Once everything is integrated, this will dry out to some extent once the older versions reach EOL. But their expertise will still be required to keep real-time in shape in mainline and in the supported mainline stable kernels.

JP: So once the upstreaming activity is complete, what happens afterward?

TG: Once upstreaming is done, efforts have to be made to enable RT support for specific Linux features currently disabled on real-time enabled kernels. Also, for quite some time, there will be fallout when other things change in the kernel, and there has to be support for kernel developers who run into the constraints of RT, which they did not have to think about before. 

The latter is a crucial point for this effort. Because there needs to be a clear longer-term commitment that the people who are deeply familiar with the matter and the concepts are not going to vanish once the mainlining is done. We can’t leave everybody else with the task of wrapping their brains around it in desperation; there cannot be institutional knowledge loss with a system as critical as this. 

The lack of such a commitment would be a showstopper on the final step because we are now at the point where the notable changes are focused on the real-time only aspects rather than welcoming cleanups, improvements, and features of general value. This, in turn, circles back to the earlier question of funding and industry support — for this final step requires several years of commitment by companies using the real-time kernel.

There’s not going to be a shortage of things to work on. It’s not going to be as much as the current upstreaming effort, but as the kernel never stops changing, this will be interesting for a long time.

JP: Thank you, Thomas, for your time this morning. It’s been an illuminating discussion.

To get involved with the real-time kernel patch for Linux, please visit the PREEMPT_RT wiki at The Linux Foundation or email real-time-membership@linuxfoundation.org

The post In the trenches with Thomas Gleixner, real-time Linux kernel patch set appeared first on Linux.com.

The post KubeEdge: Reliable Connectivity Between The Cloud & Edge appeared first on Linux.com.