Streaming data observability startup Datorios Ltd. today announced the immediate availability of a new real-time observability platform for big data processing framework Apache Flink.
With this new platform, businesses will benefit from what the startup claims is unprecedented insight into streaming data processing. These insights could help create new real-time artificial intelligence systems that can be fully audited to ensure no fraudulent activity is occurring, the startup says.
Datorios founders apply years of experience in research and development of real-time military information systems to the development of new products, with the ultimate goal of helping developers create more advanced AI applications that can operate in real-time. It states that this is true. The basic idea is that Datorios provides greater visibility into the information fed into real-time AI systems using Apache Flink-based data pipelines.
The company says it can be thought of as a “flight recorder for real-time AI.” The startup believes that increasing visibility into real-time streaming data can simplify tasks such as troubleshooting and auditing real-time AI applications.
Apache Flink has emerged as an important tool for creating real-time AI. It is an open-source big data processing framework widely used in industries such as video streaming, e-commerce, and financial services. There are use cases in data analytics, transaction processing, and AI models that require data to be processed the moment it is created and to process up-to-date information.
The Apache Flink framework is often compared to Apache Spark, which also processes information in real time. However, Flink's main advantage is its ability to perform calculations while processing data in motion, resulting in what its creators call true real-time data processing. This is different from Apache Spark. Apache Spark processes data in small chunks using a technique known as “micro-batching,” where information must be accumulated in small batches before being computed. The microbatching approach introduces slightly higher latency, which can be problematic for some types of AI systems.
Fix real-time data interruptions
Real-time AI systems rely on a continuous stream of up-to-date information to automate processes such as fraud detection, product recommendation engines, and vehicle routing. However, while Apache Flink's real-time information processing capabilities are virtually unmatched, the streaming data that Apache Flink processes is still subject to interruptions. For example, they may arrive in the wrong order, arrive multiple times, or arrive in an unexpected format. When this happens, it can create major problems for AI systems that need information to be accurate and readily accessible.
Datorios aims to solve these problems by increasing visibility into these real-time data streams, allowing developers to ensure that their AI systems are properly processing the information they need. .
More specifically, the company said its platform provides insight into event visualization, showing the flow of data through each processing operation. It also features event search functionality, allowing users to track specific events for auditing and compliance purposes. Its state monitoring feature allows users to see how the state of data changes during each processing step, while window analysis tools provide insight into areas such as the order in which data is processed, delays, and discarded records. Gain insight.
By offering these new data observability features, Datorios says it is carving out a unique niche within the broader Apache Flink ecosystem as the only tool that can identify interruptions in streaming data. .
Ronen Korman, co-founder and CEO of Datrios, said today's most innovative companies are eager to use AI to run their business on autopilot, and to do so they need to He stated that it is necessary to ensure the accuracy and reliability of the data being moved. “Datorios is their flight data recorder, showing you exactly how and why the system is behaving at any given time,” Corman explained. “Just like with aircraft, unforeseen circumstances can impact performance. The insights from Datorios make that clear and allow us to continually improve our AI.”
Datorios said it is making its platform available for free with almost no restrictions to new users to demonstrate how it can support its real-time AI systems.
Image: Datrios
Your upvote is important to us and helps us keep our content free.
Your one click below will support our mission of providing free, deep and relevant content.
Join our community on YouTube
A community of over 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies Founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many other celebrities and experts. Please join us.
thank you