Abstract: RisingWave is an open-source streaming database designed from scratch for the cloud. It implemented a Snowflake-style storage-compute separation architecture to reduce performance cost, and provides users with a PostgreSQL-like experience for stream processing. Over the last three years, RisingWave has evolved from a one-person project to a rapidly-growing product deployed by nearly 100 enterprises and startups. But the journey of building RisingWave is full of challenges. In this talk, I'd like to share with you lessons we've gained from four dimensions: 1) the decoupled compute-storage architecture, 2) the balances between stream processing and OLAP, 3) the Rust ecosystem, and 4) the product positioning. I will dive deep into technical details and then share with you my views on the future of stream processing.
talk-data.com
Topic
Rust
104
tagged
Activity Trend
Top Events
In this episode, Conor and Bryce interview Richard Feldman about the Roc programming language, what qualifies a language as a functional programming language (and whether Rust makes the cut) and more! Link to Episode 157 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachAbout the Guest: Richard Feldman is the creator of the Roc programming language, the host of the Software Unscripted podcast, and the author of Elm in Action from Manning Publications. He teaches online courses on Frontend Masters: Introduction to Rust, Introduction to Elm, and Advanced Elm. Outside of programming, he’s a fan of strategy games, heavy metal, powerlifting, and puns!
Show Notes
Date Recorded: 2023-11-13 Date Released: 2023-11-24 Software Unscripted PodcastThe Roc LanguageHaskell LanguageElixir LanguageSoftware Unscripted: Gradual vs Static Typing with Jose ValimInterview with Senior Rust Developer in 2023 (YouTube Video)Rust Iterator::mapRust Iterator::filterZig LanguageThe Essence of Functional Programming by Richard Feldman #FnConf 2022Rank-N Types (Higher Rank Types)Intro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
In this episode, Conor and Bryce interview Richard Feldman, creator of the Roc programming language, about the last edition of the Strange Loop conference, virtual vs in-person events and more. Link to Episode 156 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachAbout the Guest: Richard Feldman is the creator of the Roc programming language, the host of the Software Unscripted podcast, and the author of Elm in Action from Manning Publications. He teaches online courses on Frontend Masters: Introduction to Rust, Introduction to Elm, and Advanced Elm. Outside of programming, he’s a fan of strategy games, heavy metal, powerlifting, and puns!
Show Notes
Date Recorded: 2023-11-13 Date Released: 2023-11-17 Software Unscripted PodcastWhy Isn’t Functional Programming the Norm? – Richard FeldmanStrange Loop Conference“The Economics of Programming Languages” by Evan Czaplicki (Strange Loop 2023)“Software & The Game of Go” by David Nolen (Strange Loop 2023)“A Long Strange Loop” by Alex Miller (Strange Loop 2023)Intro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
In this episode, Conor and Bryce respond to several comments about ADSP Episode 150: Is C++ Dying? and chat about why C is the lingua franca of FFI target languages. Link to Episode 151 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachShow Notes Date Recorded: 2023-10-09 Date Released: 2023-10-13 ADSP Episode 150: Is C++ Dying?CppCast Episode 367 - SoagenOxide and Friends Episode 44 - Books in the box reduxOxide and Friends PodcastBryan Cantrill on TwitterAdam Levanthal on Twitter@ahl Punching Down TweetAlexander Bandukwala Why Do You Want to Save C++ TweetHylo Programming LanguageCircle C++ CompilerCarbon Programming LanguageCppFrontADSP Episode 137: Sean Parent on Val (vs Rust)!ADSP Episode 138: Sean Parent on Val! (Part 2)Ada LanguageSmalltalk LanguagePascal LanguageZig LanguageIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
In this episode, Conor and Bryce chat about CityStrides.com, graph algorithms and more! Link to Episode 149 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachShow Notes Date Recorded: 2023-09-25 Date Released: 2023-09-29 Hana Dusíková on TwitterHana’s co_curlNDC TechtownPeter principleADSP Episode 137: Sean Parent on Val (vs Rust)!CityStrides.comOpen Street MapsOverPasscity-strides-hacking GitHub RepoThrust Parallel Algorithm LibraryElon Musk by Ashlee VanceElon Musk by Walter IsaacsonEpisode 143 Comment About R |>Episode 142 Comment About Rust charsIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
In this episode, Conor and Bryce record live from Slovenia while driving and review Lake Bled and Bled Cream Cake and solve one problem in Haskell, Rust and C++! Link to Episode 142 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachShow Notes Date Recorded: 2023-06-18 Date Released: 2023-08-11 Lake BledBled Cream CakeMastermind Board GameGašper Ažman on TwitterRamanujan Numbers“Point-Free or Die: Tacit Programming in Haskell and Beyond” by Amar ShahLambdaDays 2023: Composition Intuition - Conor HoekstraC++17 std::transform_reduceC++98 std::inner_productC++98 std::equal_toC++98 std::equalC++98 std::plusHaskell zipWithHaskell fromEnumHaskell sumBQN LanguageAPL LanguageexactMatches Tweet from Composition Intuition Talkexact_matches Rust TweetRust Iterator traitRust str::chars()Intro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
Rust is a unique language whose traits make it very appealing for data engineering. In this session, we'll walk through the different aspects of the language that make it such a good fit for big data processing including: how it improves performance and how it provides greater safety guarantees and compatibility with a wide range of existing tools that make it well positioned to become a major building block for the future of analytics.
We will also take a hands-on look through real code examples at a few emerging technologies built on top of Rust that utilize these capabilities, and learn how to apply them to our modern lakehouse architecture.
Talk by: Oz Katz
Here’s more to explore: Why the Data Lakehouse Is Your next Data Warehouse: https://dbricks.co/3Pt5unq Lakehouse Fundamentals Training: https://dbricks.co/44ancQs
Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc
Since the release of Delta 2.0, the project has been growing at a breakneck speed. In this session, we will cover all the latest capabilities that makes Delta Lake the best format for the lakehouse. Based on lessons learned from this past year, we will introduce Project Aqueduct and how we will simplify building Delta Lake APIs from Rust and Go to Trino, Flink, and PySpark.
Talk by: Tathagata Das and Denny Lee
Connect with us: Website: https://databricks.com Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc Facebook: https://www.facebook.com/databricksinc
In this episode, Conor and Bryce close out their conversation with Sean Parent and learn why he left Apple to join Adobe. Link to Episode 139 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)How To Get Involved With Val DM Sean on TwitterVal Lang on GitHubVal Teams MeetingClick here to join the meetingMeeting ID: 298 158 296 273Passcode: D2beKFWhen: Tues/Thurs 12:30-1:00 PSTVal SlackTwitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachAbout the Guest: Sean Parent is a senior principal scientist and software architect managing Adobe’s Software Technology Lab. Sean first joined Adobe in 1993 working on Photoshop and is one of the creators of Photoshop Mobile, Lightroom Mobile, and Lightroom Web. In 2009 Sean spent a year at Google working on Chrome OS before returning to Adobe. From 1988 through 1993 Sean worked at Apple, where he was part of the system software team that developed the technologies allowing Apple’s successful transition to PowerPC.
Show Notes
Date Recorded: 2023-06-29 Date Released: 2023-07-21 ADSP Episode 137: Sean Parent on Val (vs Rust)!ADSP Episode 138: Sean Parent on Val! (Part 2)C++ On Sea ConferenceAll Sean Parent ADSP EpisodesAdobe Software Technology LabADSP Episode 40: Star Trek vs PowerPC (with Sean Parent)PostScriptJohn WarnockCharles (Chuck) GeschkeSean Parent photo with John WarnockIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
In this episode, Conor and Bryce continue their interview with Sean Parent live from C++ On Sea 2023 about the Val programming language! Link to Episode 138 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachAbout the Guest: Sean Parent is a senior principal scientist and software architect managing Adobe’s Software Technology Lab. Sean first joined Adobe in 1993 working on Photoshop and is one of the creators of Photoshop Mobile, Lightroom Mobile, and Lightroom Web. In 2009 Sean spent a year at Google working on Chrome OS before returning to Adobe. From 1988 through 1993 Sean worked at Apple, where he was part of the system software team that developed the technologies allowing Apple’s successful transition to PowerPC.
How To Get Involved With Val DM Sean on TwitterVal Lang on GitHubVal Teams MeetingClick here to join the meetingMeeting ID: 298 158 296 273Passcode: D2beKFWhen: Tues/Thurs 12:30-1:00 PSTVal SlackShow Notes Date Recorded: 2023-06-29 Date Released: 2023-07-14 ADSP Episode 137: Sean Parent on Val (vs Rust)!C++ On Sea ConferenceAll Sean Parent ADSP EpisodesAdobe Software Technology LabConor Hoekstra - Concepts vs Typeclasses vs Traits vs Protocols - Meeting C++ 2020Programming Languages Virtual MeetupThe Val Programming LanguageThe Rust Programming LanguageThe Swift Programming LanguageHalide LanguageADSP Dave Abrahams EpisodesCircle CompilerJakt Programming LanguageCppCast Episode 355 - Carbon, with Richard SmithC++ on Sea 2023: Keynote: All the Safeties - Sean ParentRust iterx libraryThe Carbon Programming LanguageIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
In this episode, Conor and Bryce interview Sean Parent live from C++ On Sea 2023 about the Val programming language and how it compares to Rust, Swift and C++. Link to Episode 137 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachAbout the Guest: Sean Parent is a senior principal scientist and software architect managing Adobe’s Software Technology Lab. Sean first joined Adobe in 1993 working on Photoshop and is one of the creators of Photoshop Mobile, Lightroom Mobile, and Lightroom Web. In 2009 Sean spent a year at Google working on Chrome OS before returning to Adobe. From 1988 through 1993 Sean worked at Apple, where he was part of the system software team that developed the technologies allowing Apple’s successful transition to PowerPC. Show Notes Date Recorded: 2023-06-29 Date Released: 2023-07-07 C++ On Sea ConferenceAll Sean Parent ADSP EpisodesAdobe Software Technology LabASL LibrariesThe Val Programming LanguageThe Rust Programming LanguageThe Swift Programming LanguageMutable Value SemanticsLLVMRust TraitsCppNorth 2022 Keynote: The Tragedy of C++, Parts One & Two - Sean ParentC++ Seasoning - Sean ParentSean Parent: “Now What? A vignette in three parts” - C++Now 2012Adobe ASL Adam & Eve ArchitectureHalide LanguageIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
In this episode, Conor and Bryce chat about the Rust Conf drama and other upcoming conferences. Link to Episode 132 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachShow Notes Date Recorded: 2023-05-31 Date Released: 2023-06-02 Programming Languages Festival (Feb 2024) KickStarterRustConfRustConf Drama (ThePhd’s Post)ThePhd on TwitterJT’s PostCppNorthRust FoundationRust Trademark DebacleCrabLangGCC Front-End For RustCrystal LanguageLambdaDays 2023Italian C++ 2023HaskellElixirBryce’s Haskell TweetArrayCast Episode on KX ConC++NowIntro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
ABOUT THE TALK: A streaming database is a potentially intimidating product to build. Frank McSherry, Chief Scientist at Materialize, breaks down the manageable parts, through three foundational choices that fit together well. Frank also talks about the trade-offs, and how their simplifications lead to a much more manageable streaming database.
ABOUT THE SPEAKER: Frank McSherry is Chief Scientist at Materialize, where he (and others) convert SQL into scale-out, streaming, and interactive dataflows. Before this, he developed the timely and differential dataflow Rust libraries (with colleagues at ETHZ), and led the Naiad research project and co-invented differential privacy while at MSR Silicon Valley. He has a PhD in computer science from the University of Washington.
ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.
Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.
FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/
ABOUT THE TALK: In this talk, Erik Bernhardsson will share how Modal starts 1000s of large containers in seconds, and what they had to do under the surface to build this. This includes a custom file system written in Rust, their own container runtime, and their own container image builder. This talk will give you an idea of how containers work along with some of the low-level Linux details underneath. We'll also talk about many infrastructure tools hold data teams back, and why they deserve faster and better tools.
ABOUT THE SPEAKER: Erik Bernhardsson is the founder and CEO of Modal, which is an infrastructure provider for data teams. Before Modal, Erik was the CTO at Better for six years, and previously spent seven years at Spotify, building the music recommendation system and running data teams.
ABOUT DATA COUNCIL: Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers.
Make sure to subscribe to our channel for the most up-to-date talks from technical professionals on data related topics including data infrastructure, data engineering, ML systems, analytics and AI from top startups and tech companies.
FOLLOW DATA COUNCIL: Twitter: https://twitter.com/DataCouncilAI LinkedIn: https://www.linkedin.com/company/datacouncil-ai/
In this episode, Conor and Bryce chat with Tristan Brindle about his new library Flux and his predecessor library Flow. Link to Episode 126 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)Twitter ADSP: The PodcastConor HoekstraBryce Adelstein LelbachAbout the Guest Tristan Brindle a freelance programmer and trainer based in London, mostly focussing on C++. He is a member of the UK national body (BSI) and ISO WG21. Occasionally I can be found at C++ conferences. He is also a director of C++ London Uni, a not-for-profit organisation offering free beginner programming classes in London and online. He has a few fun projects on GitHub that you can find out about here.
Show Notes
Date Recorded: 2023-04-05 Date Released: 2023-04-21 ADSP Episode 125: NanoRange with Tristan BrindleKeynote: Iterators and Ranges: Comparing C++ to D, Rust, and Others - Barry Revzin - CPPP 2021Rust IteratorsFlowFluxSwift SequencesEpisode 124: Vectorizing std::views::filterC++ std::find_ifC++17 std::reduceC++ std::accumulateCppCon 2016: Ben Deane “std::accumulate: Exploring an Algorithmic Empire”Intro Song Info Miss You by Sarah Jansen https://soundcloud.com/sarahjansenmusic Creative Commons — Attribution 3.0 Unported — CC BY 3.0 Free Download / Stream: http://bit.ly/l-miss-you Music promoted by Audio Library https://youtu.be/iYYxnasvfx8
The Python data ecosystem has matured during the last decade and there are less and less reasons to rely only large batch process executed in a Spark cluster, but with every large ecosystem, putting together the key pieces of technology takes some effort. There are now better storage technologies, streaming execution engines, query planners, and low level compute libraries. And modern hardware is way more powerful than what you'd probably expect. In this workshop we will explore some global-warming-reducing techniques to build more efficient data transformation pipelines in Python, and a little bit of Rust.
Writing efficient data pipelines in Python can be tricky. The standard recommendation is to use vectorized functions implemented in Numpy, Pandas, or the like. However, what to do, when the processing task does not fit these libraries? Using plain Python for processing can result in lacking performance, in particular when handling large data sets.
Rust is a modern, performance-oriented programming language that is already widely used by the Python community. Augmenting data processing steps with Rust can result in substantial speed ups. In this talk will present strategies of using Rust in a larger Python data processing pipeline with a particular focus on pragmatism and minimizing integration efforts.
Python is a very expressive and powerful language, but it is not always the fastest option for performance-critical parts of an application. Rust, on the other hand, is known for its lightning-fast runtime and low-level control, making it an attractive option for speeding up performance-sensitive portions of Python programs.
In this talk, we will present a case study of using Rust to speed up a critical component of a Python application. We will cover the following topics:
- An overview of Rust and its benefits for Python developers
- Profiling and identifying performance bottlenecks in Python application
- Implementing a solution in Rust and integrating it with the Python application using PyO3
- Measuring the performance improvements and comparing them to other optimization techniques
Attendees will learn about the potential for using Rust to boost the performance of their Python programs and how to go about doing so in their own projects.
In this talk, we will explore the use of Python's typing.Protocol, Scala's Typeclasses, and Rust's Traits.
They all offer a very powerful & elegant mechanism for abstracting over various concepts (such as Serialization) in a modular manner.
We will compare and contrast the syntax and implementation of these constructs in each language and discuss their strengths and weaknesses. We will also look at real-world examples of how these features are used in each language to specify behavior, and consider differences in terms of type system expressiveness and effectiveness. By the end of the talk, attendees will have a better understanding of the differences and similarities between these three language features, and will be able to make informed decisions about which one is best suited for their needs.
In this talk, we will report on our experiences switching from Pandas to Polars in a real-world ML project. Polars is a new high-performance dataframe library for Python based on Apache Arrow and written in Rust. We will compare the performance of polars with the popular pandas library, and show how polars can provide significant speed improvements for data manipulation and analysis tasks. We will also discuss the unique features of polars, such as its ability to handle large datasets that do not fit into memory, and how it feels in practice to make the switch from Pandas. This talk is aimed at data scientists, analysts, and anyone interested in fast and efficient data processing in Python.