This book is a collective work by a number of leading scientists, analysts, engineers, mathematicians and statisticians who have been working at the forefront of data analysis and related applications, arising from data science, operations research, engineering, machine learning or statistics. The chapters of this collaborative work represent a cross-section of current research interests in the above scientific areas. The collected material has been divided into appropriate sections to provide the reader with both theoretical and applied information on data analysis methods, models and techniques, along with appropriate applications. Data Analysis and Related Applications 4 investigates a number of different topics in the areas mentioned above, touching on statistical analysis, stochastic processes, estimation methods, algorithms, distributions and networks, among others.

Join Kirk and Wes as they delve into the remarkable journey of Wes Cummins, the visionary founder and CEO of Applied Digital. Wes recounts his evolution from a childhood spent on a family potato farm in Southern Idaho to spearheading innovations in next-generation digital infrastructure. He discusses his early fascination with technology, which guided him from academics and athletics to a career in finance and tech. The conversation highlights Wes’s pivot from Bitcoin mining to building high-performance data centers for AI workloads, especially in North America. Wes emphasizes the advantages of operating in smaller towns, like those in North Dakota, where Applied Digital contributes to local economies and creates high-paying jobs. As they explore the critical role of infrastructure in the evolving tech landscape, Wes paints a picture of a future where Applied Digital plays a key role in the global AI race, underscoring the importance of sustainable, community-focused innovation.

Join Kirk and Wes as they delve into the remarkable journey of Wes Cummins, the visionary founder and CEO of Applied Digital. Wes recounts his evolution from a childhood spent on a family potato farm in Southern Idaho to spearheading innovations in next-generation digital infrastructure. He discusses his early fascination with technology, which guided him from academics and athletics to a career in finance and tech. The conversation highlights Wes’s pivot from Bitcoin mining to building high-performance data centers for AI workloads, especially in North America. Wes emphasizes the advantages of operating in smaller towns, like those in North Dakota, where Applied Digital contributes to local economies and creates high-paying jobs. As they explore the critical role of infrastructure in the evolving tech landscape, Wes paints a picture of a future where Applied Digital plays a key role in the global AI race, underscoring the importance of sustainable, community-focused innovation.

For more about us: https://linktr.ee/overwatchmissioncritical

Building and managing AI products comes with its own set of unique challenges. Especially when they are under intense scrutiny like mobile and home assistants have dealt with in recent years. From dealing with the unpredictable nature of machine learning models to ensuring that your product is both ethical and user-friendly, the path to success isn’t always clear. But how do you navigate these complexities and still deliver a product that meets business goals? What key steps can you take to align AI innovation with measurable outcomes and long-term success? Marily Nika is one of the world's leading thinkers on product management for artificial intelligence. At Google, she manages the generative AI product features for Google Assistant. Marily also founded AI Product Academy, where she runs a BootCamp on AI product management, and she teaches the subject on Maven. Previously, Marily was an AI Product Lead in Meta's Reality Labs, and the AI Product Lead for Google Glass. She is also an Executive Fellow at Harvard Business School. In the episode, Richie and Marily explore the unique challenges of AI product management, experimentation, ethical considerations in AI product management, collaboration, skills needed to succeed in AI product development, the career path to work in AI as a Product Manager, key metrics for AI products and much more.  Links Mentioned in the Show: Komo AIConnect with MarilyMarily’s Course: AI Product Management Bootcamp with CertificationSkill Track: AI Business FundamentalsRelated Episode: Building Human-Centered AI Experiences with Haris Butt, Head of Product Design at ClickUpRewatch sessions from RADAR: AI Edition New to DataCamp? Learn on the go using the DataCamp mobile appEmpower your business with world-class data and AI skills with DataCamp for business

A set of divergences in the global economy raise tail risks to the expansion. While we see numerous reasons for a soft-landing, next week’s PMIs and US labor market news are likely to exacerbate the tensions. Despite this, US growth is robust while Europe is held back by a weak Germany. China’s policy shift this week is encouraging only if a signal of more to come. Joe shares pics from his trip to the North Branch Outing Club in Northern Michigan.

Speakers:

Bruce Kasman

Joseph Lupton

This podcast was recorded on 27 September 2024.

This communication is provided for information purposes only. Institutional clients please visit www.jpmm.com/research/disclosures for important disclosures. © 2024 JPMorgan Chase & Co. All rights reserved. This material or any portion hereof may not be reprinted, sold or redistributed without the written consent of J.P. Morgan. It is strictly prohibited to use or share without prior written consent from J.P. Morgan any research material received from J.P. Morgan or an authorized third-party (“J.P. Morgan Data”) in any third-party artificial intelligence (“AI”) systems or models when such J.P. Morgan Data is accessible by a third-party. It is permissible to use J.P. Morgan Data for internal business purposes only in an AI system or model that protects the confidentiality of J.P. Morgan Data so as to prevent any and all access to or use of such J.P. Morgan Data by any third-party.

This book covers modern data engineering functions and important Python libraries, to help you develop state-of-the-art ML pipelines and integration code. The book begins by explaining data analytics and transformation, delving into the Pandas library, its capabilities, and nuances. It then explores emerging libraries such as Polars and CuDF, providing insights into GPU-based computing and cutting-edge data manipulation techniques. The text discusses the importance of data validation in engineering processes, introducing tools such as Great Expectations and Pandera to ensure data quality and reliability. The book delves into API design and development, with a specific focus on leveraging the power of FastAPI. It covers authentication, authorization, and real-world applications, enabling you to construct efficient and secure APIs using FastAPI. Also explored is concurrency in data engineering, examining Dask's capabilities from basic setup to crafting advanced machine learning pipelines. The book includes development and delivery of data engineering pipelines using leading cloud platforms such as AWS, Google Cloud, and Microsoft Azure. The concluding chapters concentrate on real-time and streaming data engineering pipelines, emphasizing Apache Kafka and workflow orchestration in data engineering. Workflow tools such as Airflow and Prefect are introduced to seamlessly manage and automate complex data workflows. What sets this book apart is its blend of theoretical knowledge and practical application, a structured path from basic to advanced concepts, and insights into using state-of-the-art tools. With this book, you gain access to cutting-edge techniques and insights that are reshaping the industry. This book is not just an educational tool. It is a career catalyst, and an investment in your future as a data engineering expert, poised to meet the challenges of today's data-driven world. What You Will Learn Elevate your data wrangling jobs by utilizing the power of both CPU and GPU computing, and learn to process data using Pandas 2.0, Polars, and CuDF at unprecedented speeds Design data validation pipelines, construct efficient data service APIs, develop real-time streaming pipelines and master the art of workflow orchestration to streamline your engineering projects Leverage concurrent programming to develop machine learning pipelines and get hands-on experience in development and deployment of machine learning pipelines across AWS, GCP, and Azure Who This Book Is For Data analysts, data engineers, data scientists, machine learning engineers, and MLOps specialists

Learn how to leverage the scientific computing and data analysis capabilities of Python, its standard library, and popular open-source numerical Python packages like NumPy, SymPy, SciPy, matplotlib, and more. This book demonstrates how to work with mathematical modeling and solve problems with numerical, symbolic, and visualization techniques. It explores applications in science, engineering, data analytics, and more. Numerical Python, Third Edition, presents many case study examples of applications in fundamental scientific computing disciplines, as well as in data science and statistics. This fully revised edition, updated for each library's latest version, demonstrates Python's power for rapid development and exploratory computing due to its simple and high-level syntax and many powerful libraries and tools for computation and data analysis. After reading this book, readers will be familiar with many computing techniques, including array-based and symbolic computing, visualization and numerical file I/O, equation solving, optimization, interpolation and integration, and domain-specific computational problems, such as differential equation solving, data analysis, statistical modeling, and machine learning. What You'll Learn Work with vectors and matrices using NumPy Review Symbolic computing with SymPy Plot and visualize data with Matplotlib Perform data analysis tasks with Pandas and SciPy Understand statistical modeling and machine learning with statsmodels and scikit-learn Optimize Python code using Numba and Cython Who This Book Is For Developers who want to understand how to use Python and its ecosystem of libraries for scientific computing and data analysis.

Machine Learning practitioners build predictive models from "noisy" data resulting in uncertain predictions. But what does "noise" mean in a machine learning context?

Send us a text Welcome to the cozy corner of the tech world where ones and zeros mingle with casual chit-chat. Datatopics Unplugged is your go-to spot for relaxed discussions around tech, news, data, and society. We dive into conversations smoother than your morning coffee (but let’s be honest, just as caffeinated) where industry insights meet light-hearted banter. Whether you’re a data wizard or just curious about the digital chaos around us, kick back and get ready to talk shop—unplugged style! In this episode: Farewell Pandas, Hello Future: Pandas is out, and Ibis is in. We're talking faster, smarter data processing—featuring the rise of DuckDB and the powerhouse that is Polars. Is this the end of an era for Pandas?UV vs. Rye: Forget pip—are these new Python package managers built in Rust the future? We break down UV, Rye, and what it all means for your next Python project.AI-Generated Podcasts: Is AI about to take over your favorite podcasts? We explore the potential of Google’s Notebook LM to transform content into audio gold.When AI Steals Your Voice: Jeff Geerling’s voice gets cloned by AI—without his consent. We dive into the wild world of voice cloning, the ethics, and the future of AI-generated media.Hacking AI with Prompt Injection: Could you outsmart AI? We share some wild strategies from the game Gandalf that challenge your prompt injection skills and teach you how to jailbreak even the toughest guardrails.Jony Ive’s New Gadget Rumor: Is Jony Ive plotting an Apple killer? Rumors are swirling about a new AI-powered handheld device that could shake up the smartphone market.Zero-Downtime Deployments with Kamal Proxy: No more downtime! We geek out over Kamal Proxy, the sleek HTTP tool designed for effortless Docker deployments.Function Calling and LLMs: Get ready for the next evolution in AI—function calling. We discuss its rise in LLMs and dive into the Gorilla project, the leaderboard testing the future of smart APIs.

Scaling machine learning at large organizations like Renault Group presents unique challenges, in terms of scales, legal requirements, and diversity of use cases. Data scientists require streamlined workflows and automated processes to efficiently deploy models into production. We present an MLOps pipeline based on python Kubeflow and GCP Vertex AI API designed specifically for this purpose. It enables data scientists to focus on code development for pre-processing, training, evaluation, and prediction. This MLOPS pipeline is a cornerstone of the AI@Scale program, which aims to roll out AI across the Group.

We choose a Python-first approach, allowing Data scientists to focus purely on writing preprocessing or ML oriented Python code, also allowing data retrieval through SQL queries. The pipeline addresses key questions such as prediction type (batch or API), model versioning, resource allocation, drift monitoring, and alert generation. It favors faster time to market with automated deployment and infrastructure management. Although we encountered pitfalls and design difficulties, that we will discuss during the presentation, this pipeline integrates with a CI/CD process, ensuring efficient and automated model deployment and serving.

Finally, this MLOps solution empowers Renault data scientists to seamlessly translate innovative models into production, and smoothen the development of scalable, and impactful AI-driven solutions.

Rising concerns over IT's carbon footprint necessitate tools that gauge and mitigate these impacts. This session introduces CodeCarbon, an open-source tool that estimates computing's carbon emissions by measuring energy use across hardware components. Aimed at AI researchers and data scientists, CodeCarbon provides actionable insights into the environmental costs of computational projects, supporting efforts towards sustainability without requiring deep technical expertise.

This talk from the main contributors of Code Carbon will cover the environmental impact of IT, the possibilities to estimate it and a demo of CodeCarbon.

The MedTech industry is undergoing a revolutionary transformation with continuous innovations promising greater precision, efficiency, and accessibility. In particular oncology, a branch of medicine that focuses on cancer, will benefit immensely from these new technologies, which may enable clinicians to detect cancer earlier and increase chances of survival. Detecting cancerous cells in microscopic photography of cells (Whole Slide Images, aka WSIs) is usually done with segmentation algorithms, which neural networks (NNs) are very good at. While using ML and NNs for image segmentation is a fairly standard task with established solutions, doing it on WSIs is a different kettle of fish. Most training pipelines and systems have been designed for analytics, meaning huge columns of small individual datums. In the case of WSIs, a single image is so huge that its file can be up to dozens of gigabytes. To allow innovation in medical imaging with AI, we need efficient and affordable ways to store and process these WSIs at scale.

MAPIE (Model Agnostic Prediction Interval Estimator) is your go-to solution for managing uncertainties and risks in machine learning models. This Python library, nestled within scikit-learn-contrib, offers a way to calculate prediction intervals with controlled coverage rates for regression, classification, and even time series analysis. But it doesn't stop there - MAPIE can also be used to handle more complex tasks like multi-label classification and semantic segmentation in computer vision, ensuring probabilistic guarantees on crucial metrics like recall and precision. MAPIE can be integrated with any model - whether it's scikit-learn, TensorFlow, or PyTorch. Join us as we delve into the world of conformal predictions and how to quickly manage your uncertainties using MAPIE.

Link to Github: https://github.com/scikit-learn-contrib/MAPIE

Jordan Morrow has written a ton, including four books. We chat about the process of writing books, the ins and outs of working with a publisher, the role of AI in writing, and much more. If you're interested in writing a book, this is a crash course in what you should know. Enjoy!

Every organization today is exploring generative AI to drive value and push their business forward. But a common pitfall is that AI strategies often don’t align with business objectives, leading companies to chase flashy tools rather than focusing on what truly matters. How can you avoid these traps and ensure your AI efforts are not only innovative but also aligned with real business value?  Leon Gordon, is a leader in data analytics and AI. A current Microsoft Data Platform MVP based in the UK, founder of Onyx Data. During the last decade, he has helped organizations improve their business performance, use data more intelligently, and understand the implications of new technologies such as artificial intelligence and big data. Leon is an Executive Contributor to Brainz Magazine, a Thought Leader in Data Science for the Global AI Hub, chair for the Microsoft Power BI – UK community group and the DataDNA data visualization community as well as an international speaker and advisor. In the episode, Adel and Leon explore aligning AI with business strategy, building AI use-cases, enterprise AI-agents, AI and data governance, data-driven decision making, key skills for cross-functional teams, AI for automation and augmentation, privacy and AI and much more.  Links Mentioned in the Show: Onyx DataConnect with LeonLeon’s Linkedin Course - How to Build and Execute a Successful Data StrategySkill Track: AI Business FundamentalsRelated Episode: Generative AI in the Enterprise with Steve Holden, Senior Vice President and Head of Single-Family Analytics at Fannie MaeRewatch sessions from RADAR: AI Edition New to DataCamp? Learn on the go using the DataCamp mobile appEmpower your business with world-class data and AI skills with DataCamp for business

Discover metalearners, a cutting-edge Python library designed for Causal Inference with particularly flexible and user-friendly MetaLearner implementations. metalearners leverages the power of conventional Machine Learning estimators and molds them into causal treatment effect estimators. This talk is targeted towards data professionals with some Python and Machine Learning competences, guiding them to optimizing interventions such as 'Which potential customers should receive a voucher to optimally allocate a voucher budget?' or 'Which patients should receive which medical treatment?' based on causal interpretations.

sktime is a widely used scikit-learn compatible library for learning with time series. sktime is easily extensible by anyone, and interoperable with the pydata/numfocus stack.

This talk presents progress, challenges, and newest features off the press, in extending the sktime framework to deep learning and foundation models.

Recent progress in generative AI and deep learning is leading to an ever-exploding number of popular “next generation AI” models for time series tasks like forecasting, classification, segmentation.

Particular challenges of the new AI ecosystem are inconsistent formal interfaces, different deep learning backends, vendor specific APIs and architectures which do not match sklearn-like patterns well – every practitioner who has tried to use at least two such models at the same time (outside sktime) will have their individual painful memories.

We show how sktime brings its unified interface architecture for time series modelling to the brave new AI frontier, using novel design patterns building on ideas from hugging face and scikit-learn, to provide modular, extensible building blocks with a simple specification language.

Understanding the effectiveness of various marketing channels is crucial to maximise the return on investment (ROI). However, the limitation of third-party cookies and an ever-growing focus on privacy make it difficult to rely on basic analytics. This talk discusses a pioneering project where a Bayesian model was employed to assess the marketing media mix effectiveness of WeRoad, the fastest-growing Italian tour operator.

The Bayesian approach allows for the incorporation of prior knowledge, seamlessly updating it with new data to provide robust, actionable insights. This project leveraged a Bayesian model to unravel the complex interactions between marketing channels such as online ads, social media, and promotions. We'll dive deep into how the Bayesian model was designed, discussing how we provided the AI system with expert knowledge, and presenting how delays and saturation were modelled.

We will also tackle aspects of the technical implementation, discussing how Python, PyMC, and Streamlit provided us with the all the tools we needed to develop an effective, efficient, and user-friendly system.

Attendees will walk away with:

• A simple understanding of the Bayesian approach and why it matters.
• Concrete examples of the transformative impact on WeRoad's marketing strategy.
• A blueprint to harness predictive models in their business strategies.

In the rapidly evolving landscape of Artificial Intelligence (AI), open source and openness AI have emerged as crucial factors in fostering innovation, transparency, and accountability. Mistral AI's release of the open-weight Mistral 7B model has sparked significant adoption and demand, highlighting the importance of open-source and customization in building AI applications. This talk focuses on the Mistral AI model landscape, the benefits of open-source and customization, and the opportunities for building AI applications using Mistral models.

--- The GovExperts is the new mini-series from GovEx Data Points spotlighting some of the top minds in public sector data. In our inaugural episode we discuss what generative AI is good at, how cities are interacting with it, and what it means for the workforce.

--- We’re chatting with Andrew Nicklin, Senior Research Data Manager at GovEx. Andrew takes us from his early days at the NYC parks department to his pivotal role in launching the NYC Open Data platform, and how this experience led him to GovEx at the invitation of founder Beth Blauer.

Discover how cities are already using AI to power chatbots and manage documents, and why Andrew believes AI could help residents feel more comfortable accessing sensitive services like housing or food assistance. Wondering if AI will replace public sector workers? Andrew says rather than take jobs, it will most likely transform them, freeing up public servants to tackle big challenges. Plus, get an exclusive preview of GovEx’s new City Data Explorer, a tool that uses 1.7 million data points to track 40 key metrics across the 100 largest U.S. cities.  --- Learn more about GovEx --- Fill out our listener survey!

The first episode of The Pragmatic Engineer Podcast is out. Expect similar episodes every other Wednesday. You can add the podcast in your favorite podcast player, and have future episodes downloaded automatically. Listen now on Apple, Spotify, and YouTube. Brought to you by: • Codeium: ​​Join the 700K+ developers using the IT-approved AI-powered code assistant. • TLDR: Keep up with tech in 5 minutes — On the first episode of the Pragmatic Engineer Podcast, I am joined by Simon Willison. Simon is one of the best-known software engineers experimenting with LLMs to boost his own productivity: he’s been doing this for more than three years, blogging about it in the open. Simon is the creator of Datasette, an open-source tool for exploring and publishing data. He works full-time developing open-source tools for data journalism, centered on Datasette and SQLite. Previously, he was an engineering director at Eventbrite, joining through the acquisition of Lanyrd, a Y Combinator startup he co-founded in 2010. Simon is also a co-creator of the Django Web Framework. He has been blogging about web development since the early 2000s. In today’s conversation, we dive deep into the realm of Gen AI and talk about the following:  • Simon’s initial experiments with LLMs and coding tools • Why fine-tuning is generally a waste of time—and when it’s not • RAG: an overview • Interacting with GPTs voice mode • Simon’s day-to-day LLM stack • Common misconceptions about LLMs and ethical gray areas  • How Simon’s productivity has increased and his generally optimistic view on these tools • Tips, tricks, and hacks for interacting with GenAI tools • And more! I hope you enjoy this episode. — In this episode, we cover: (02:15) Welcome (05:28) Simon’s ‘scary’ experience with ChatGPT (10:58) Simon’s initial experiments with LLMs and coding tools (12:21) The languages that LLMs excel at (14:50) To start LLMs by understanding the theory, or by playing around? (16:35) Fine-tuning: what it is, and why it’s mostly a waste of time (18:03) Where fine-tuning works (18:31) RAG: an explanation (21:34) The expense of running testing on AI (23:15) Simon’s current AI stack  (29:55) Common misconceptions about using LLM tools (30:09) Simon’s stack – continued  (32:51) Learnings from running local models (33:56) The impact of Firebug and the introduction of open-source  (39:42) How Simon’s productivity has increased using LLM tools (41:55) Why most people should limit themselves to 3-4 programming languages (45:18) Addressing ethical issues and resistance to using generative AI (49:11) Are LLMs are plateauing? Is AGI overhyped? (55:45) Coding vs. professional coding, looking ahead (57:27) The importance of systems thinking for software engineers  (1:01:00) Simon’s advice for experienced engineers (1:06:29) Rapid-fire questions — Where to find Simon Willison: • X: https://x.com/simonw • LinkedIn: https://www.linkedin.com/in/simonwillison/ • Website: https://simonwillison.net/ • Mastodon: https://fedi.simonwillison.net/@simon — Referenced: • Simon’s LLM project: https://github.com/simonw/llm • Jeremy Howard’s Fast Ai: https://www.fast.ai/ • jq programming language: https://en.wikipedia.org/wiki/Jq_(programming_language) • Datasette: https://datasette.io/ • GPT Code Interpreter: https://platform.openai.com/docs/assistants/tools/code-interpreter • Open Ai Playground: https://platform.openai.com/playground/chat • Advent of Code: https://adventofcode.com/ • Rust programming language: https://www.rust-lang.org/ • Applied AI Software Engineering: RAG: https://newsletter.pragmaticengineer.com/p/rag • Claude: https://claude.ai/ • Claude 3.5 sonnet: https://www.anthropic.com/news/claude-3-5-sonnet • ChatGPT can now see, hear, and speak: https://openai.com/index/chatgpt-can-now-see-hear-and-speak/ • GitHub Copilot: https://github.com/features/copilot • What are Artifacts and how do I use them?: https://support.anthropic.com/en/articles/9487310-what-are-artifacts-and-how-do-i-use-them • Large Language Models on the command line: https://simonwillison.net/2024/Jun/17/cli-language-models/ • Llama: https://www.llama.com/ • MLC chat on the app store: https://apps.apple.com/us/app/mlc-chat/id6448482937 • Firebug: https://en.wikipedia.org/wiki/Firebug_(software)# • NPM: https://www.npmjs.com/ • Django: https://www.djangoproject.com/ • Sourceforge: https://sourceforge.net/ • CPAN: https://www.cpan.org/ • OOP: https://en.wikipedia.org/wiki/Object-oriented_programming • Prolog: https://en.wikipedia.org/wiki/Prolog • SML: https://en.wikipedia.org/wiki/Standard_ML • Stabile Diffusion: https://stability.ai/ • Chain of thought prompting: https://www.promptingguide.ai/techniques/cot • Cognition AI: https://www.cognition.ai/ • In the Race to Artificial General Intelligence, Where’s the Finish Line?: https://www.scientificamerican.com/article/what-does-artificial-general-intelligence-actually-mean/ • Black swan theory: https://en.wikipedia.org/wiki/Black_swan_theory • Copilot workspace: https://githubnext.com/projects/copilot-workspace • Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems: https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321 • Bluesky Global: https://www.blueskyglobal.org/ • The Atrocity Archives (Laundry Files #1): https://www.amazon.com/Atrocity-Archives-Laundry-Files/dp/0441013651 • Rivers of London: https://www.amazon.com/Rivers-London-Ben-Aaronovitch/dp/1625676158/ • Vanilla JavaScript: http://vanilla-js.com/ • jQuery: https://jquery.com/ • Fly.io: https://fly.io/ — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe