Kafka on Facile.it Engineering

Kafka on Facile.it Engineering https://engineering.facile.it/categories/kafka/ Recent content in Kafka on Facile.it Engineering Hugo en-us Mon, 18 Sep 2023 13:00:00 +0000 Venerdì Protetto | September edition: Code shipping, Community tools, Quix Streams Library https://engineering.facile.it/blog/eng/v-protetto8-9-2023/ Mon, 18 Sep 2023 13:00:00 +0000 https://engineering.facile.it/blog/eng/v-protetto8-9-2023/ <head> <script type="application/ld+json"> { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Venerdì Protetto | September edition: Code shipping, Community tools, Quix Streams Library", "keywords": "Code shipping, Community tools, Quix Streams Library", "wordcount": "512", "publisher": "Facile.it Engineering", "url": "https://engineering.facile.it/", "image": "https://engineering.facile.it/images/social/orange-social-preview.png", "datePublished": "2023-09-18", "dateCreated": "2023-09-12", "dateModified": "2023-09-18", "description": "Abstracts of the talks held during the Venerdì Protetto on September 8th, 2023", "articleBody": "This page contains the abstracts of the talks held during the latest Venerdì Protetto on September 8, 2023. Topics: Code shipping by Luca Florio (Spotify) Community tools by Giovanni Cardamone Quix Streams Library by Thomas Neubauer (Quix) The overview of Venerdì Protetto is available here. Code Shipping Ship it! From your laptop to production multiple times a day By Luca Florio, Spotify Shipping code We all love to ship our code to production as fast as possible. But how to ensure (well, kind of) that we are not breaking stuff? In the talk, we explored how to implement a CI/CD pipeline to bring the code from our laptops to our servers in the cloud in a safe way. We covered testing, PR review, infrastructure as a code, canary analysis, and all the safeguards that make it hard to take our service down by mistake. Oh, and we need to keep it up and running after we deploy it as well. That is important too, right? Community Tools A place where we can meet By Giovanni Cardamone, Facile.it Community tools Community Tools is an initiative to create a unified community within Facile.it. The key points are improving, having fun, and bringing value to everyone. The initiative focuses on introducing new tools to the company to build a nicer working environment and better integrate into the "Facile lifestyle". Every community member can make a difference by having a space to communicate and compare with others. Community Tools is also a place to find new stuff and learn from each other's work. New workflows, programming languages, and methodologies can be experimented with freely and without deadlines to build everything you think is missing. Quix Streams Library A Python-Kafka connector for data-intensive workloads By Thomas Neubauer, Quix Quix Streams Library This talk introduced Quix Streams, an open-source Python library for data-intensive workloads on Kafka. We discussed the unique problems that this library is designed to solve, and how it was shaped by the challenges of building a Kafka-based solution for Formula 1 cars at McLaren— a solution that needed to process a colossal firehose of sensor data coming in at thousands of samples per second. We also explained why we combined a Kafka API approach with a stream processing library and provided developers with a familiar Pandas DataFrame-like interface. We saw the library in action with a sentiment analysis demo. In this demo, we calculated sentiment scores for incoming messages in a demo chap app—all in real-time, using the HuggingFace Transformer's API. In the end, we connected to Twitter streaming API to send a high volume of data into the pipeline to simulate this use case at scale. We saw how the library can simplify tasks such as: Subscribing to topics and deserializing incoming messages into table rows. Running calculations on a rolling window of messages. Using memory states to apply different functions such as aggregation or filtering. Automatically outputting the results of calculations into downstream topics. Managing state without the hassle of checkpointing and queues. The archive of all Venerdì Protetto talks is available here. ", "author": { "@type": "Person", "name": "Ana" } } </script> </head> <p>This page contains the abstracts of the talks held during the latest Venerdì Protetto on September 8, 2023.</p> <p>Topics:</p> <ul> <li><a href="#code-shipping">Code shipping</a> by <a href="https://www.linkedin.com/in/elleflorio/">Luca Florio</a> (<a href="https://engineering.atspotify.com/">Spotify</a>)</li> <li><a href="#community-tools">Community tools</a> by <a href="https://www.linkedin.com/in/giovanni-cardamone-41306973">Giovanni Cardamone</a></li> <li><a href="#quix-streams-library">Quix Streams Library</a> by <a href="https://www.linkedin.com/in/tom%C3%A1%C5%A1-neubauer-a10bb144/">Thomas Neubauer</a> (<a href="https://quix.io/">Quix</a>)</li> </ul> <p>The overview of Venerdì Protetto is available <a href="https://engineering.facile.it/blog/eng/v-protetto/">here</a>.</p>