This IBM® Redbooks® publication demonstrates and documents how to implement and manage an IBM PowerLinux™ cluster for big data focusing on hardware management, operating systems provisioning, application provisioning, cluster readiness check, hardware, operating system, IBM InfoSphere® BigInsights™, IBM Platform Symphony®, IBM Spectrum™ Scale (formerly IBM GPFS™), applications monitoring, and performance tuning. This publication shows that IBM PowerLinux clustering solutions (hardware and software) deliver significant value to clients that need cost-effective, highly scalable, and robust solutions for big data and analytics workloads. This book documents and addresses topics on how to use IBM Platform Cluster Manager to manage PowerLinux BigData data clusters through IBM InfoSphere BigInsights, Spectrum Scale, and Platform Symphony. This book documents how to set up and manage a big data cluster on PowerLinux servers to customize application and programming solutions, and to tune applications to use IBM hardware architectures. This document uses the architectural technologies and the software solutions that are available from IBM to help solve challenging technical and business problems. This book is targeted at technical professionals (consultants, technical support staff, IT Architects, and IT Specialists) that are responsible for delivering cost-effective Linux on IBM Power Systems™ solutions that help uncover insights among client's data so they can act to optimize business results, product development, and scientific discoveries.

Today's businesses, applications, social media, and online transactions generate more data than ever before. This data can be explored and analyzed to provide tremendous business value. IBM® Watson™ Explorer and IBM InfoSphere® Master Data Management (InfoSphere MDM) enable organizations to simultaneously explore and derive insights from enterprise data that was traditionally stored in "silos" in enterprise applications, different data repositories, and in different data formats. This IBM Redbooks® publication provides information about Watson Explorer 9.0, InfoSphere MDM, and IBM InfoSphere MDM Probabilistic Matching Engine for InfoSphere BigInsights™ (PME for BigInsights). It gives you an overview, describes the architecture, and presents use cases that you can use to accomplish the following tasks: Understand the core capabilities of Watson Explorer, InfoSphere MDM, and PME for BigInsights. Realize the full potential of Watson Explorer applications. Describe the integration and value of the combination of Watson Explorer and InfoSphere MDM. Build a 360-degree information application. Learn by example by following hands-on lab scenarios.

IBM® InfoSphere® Guardium® provides the simplest, most robust solution for data security and data privacy by assuring the integrity of trusted information in your data center. InfoSphere Guardium helps you reduce support costs by automating the entire compliance auditing process across heterogeneous environments. InfoSphere Guardium offers a flexible and scalable solution to support varying customer architecture requirements. This IBM Redbooks® publication provides a guide for deploying the Guardium solutions. This book also provides a roadmap process for implementing an InfoSphere Guardium solution that is based on years of experience and best practices that were collected from various Guardium experts. We describe planning, installation, configuration, monitoring, and administrating an InfoSphere Guardium environment. We also describe use cases and how InfoSphere Guardium integrates with other IBM products. The guidance can help you successfully deploy and manage an IBM InfoSphere Guardium system. This book is intended for the system administrators and support staff who are responsible for deploying or supporting an InfoSphere Guardium environment.

This IBM® Redbooks® publication describes visual development, visualization, adapters, analytics, and accelerators for IBM InfoSphere® Streams (V3), a key component of the IBM Big Data platform. Streams was designed to analyze data in motion, and can perform analysis on incredibly high volumes with high velocity, using a wide variety of analytic functions and data types. The Visual Development environment extends Streams Studio with drag-and-drop development, provides round tripping with existing text editors, and is ideal for rapid prototyping. Adapters facilitate getting data in and out of Streams, and V3 supports WebSphere MQ, Apache Hadoop Distributed File System, and IBM InfoSphere DataStage. Significant analytics include the native Streams Processing Language, SPSS Modeler analytics, Complex Event Processing, TimeSeries Toolkit for machine learning and predictive analytics, Geospatial Toolkit for location-based applications, and Annotation Query Language for natural language processing applications. Accelerators for Social Media Analysis and Telecommunications Event Data Analysis sample programs can be modified to build production level applications. Want to learn how to analyze high volumes of streaming data or implement systems requiring high performance across nodes in a cluster? Then this book is for you. Please note that the additional material referenced in the text is not available from IBM.

Today's businesses, applications, social media, and online transactions generate more data than ever before. This data can be explored and analyzed to provide tremendous business value. IBM® InfoSphere® Data Explorer (Data Explorer) enables organizations to simultaneously explore and derive insights from enterprise data that was traditionally stored in "silos" in enterprise applications, different data repositories, and in different data formats. Applications developed using Data Explorer identify data relationships across these silos, unlocking the business value that is inherent in a unified, 360-degree view of the information related to business entities, such as application users, customers, products, and so on. This IBM Redbooks® publication provides information about InfoSphere Data Explorer 9.0. It gives an overview, describes the architecture, and presents use cases that you can use to accomplish the following tasks: Understand the core capabilities of Data Explorer Engine and Data Explorer Application Builder. Realize the full potential of Data Explorer applications. Describe the integration and value of the combination of Data Explorer and InfoSphere Master Data Management. Build a 360-degree information application. Learn by example by following a hands-on lab scenario.

This IBM® Redbooks® publication describes the architecture and components of IBM InfoSphere® Optim™ Performance Manager Extended Edition. Intended for DBAs and those involved in systems performance, it provides information for installation, configuration, and deployment. InfoSphere Optim Performance Manager delivers a new paradigm used to monitor and manage database and database application performance issues. It describes product dashboards and reports and provides scenarios for how they can be used to identify, diagnose, prevent, and resolve database performance problems. IBM InfoSphere Optim Query Workload Tuner facilitates query and query workload analysis and provides expert recommendations for improving query and query workload performance. Use InfoSphere Optim Performance Manager to identify slow running queries, top CPU consumers, or query workloads needing performance improvements and seamlessly transfer them to InfoSphere Optim Query Workload Tuner for analysis and recommendations. This is done using query formatting annotated with relevant statistics, access plan graphical or hierarchical views, and access plan analysis. It further provides recommendations for improving query structure, statistics collection, and indexes including generated command syntax and rationale for the recommendations. Please note that the additional material referenced in the text is not available from IBM.

As world activities become more integrated, the rate of data growth has been increasing exponentially. And as a result of this data explosion, current data management methods can become inadequate. People are using the term big data (sometimes referred to as Big Data) to describe this latest industry trend. IBM® is preparing the next generation of technology to meet these data management challenges. To provide the capability of incorporating big data sources and analytics of these sources, IBM developed a stream-computing product that is based on the open source computing framework Apache Hadoop. Each product in the framework provides unique capabilities to the data management environment, and further enhances the value of your data warehouse investment. In this IBM Redbooks® publication, we describe the need for big data in an organization. We then introduce IBM InfoSphere® BigInsights™ and explain how it differs from standard Hadoop. BigInsights provides a packaged Hadoop distribution, a greatly simplified installation of Hadoop and corresponding open source tools for application development, data movement, and cluster management. BigInsights also brings more options for data security, and as a component of the IBM big data platform, it provides potential integration points with the other components of the platform. A new chapter has been added to this edition. Chapter 11 describes IBM Platform Symphony®, which is a new scheduling product that works with IBM Insights, bringing low-latency scheduling and multi-tenancy to IBM InfoSphere BigInsights. The book is designed for clients, consultants, and other technical professionals.

IBM® InfoSphere® Master Data Management Reference Data Management Hub (InfoSphere MDM Ref DM Hub) is designed as a ready-to-run application that provides the governance, process, security, and audit control for managing reference data as an enterprise standard, resulting in fewer errors, reduced business risk and cost savings. This IBM Redbooks® publication describes where InfoSphere MDM Ref DM Hub fits into information management reference architecture. It explains the end-to-end process of an InfoSphere MDM Ref DM Hub implementation including the considerations of planning a reference data management project, requirements gathering and analysis, model design in detail, and integration considerations and scenarios. It then shows implementation examples and the ongoing administration tasks. This publication can help IT professionals who are interested or have a need to manage reference data efficiently and implement an InfoSphere MDM Ref DM Hub solution with ease.

There are multiple uses for big data in every industry—from analyzing larger volumes of data than was previously possible to driving more precise answers, to analyzing data at rest and data in motion to capture opportunities that were previously lost. A big data platform will enable your organization to tackle complex problems that previously could not be solved using traditional infrastructure. As the amount of data available to enterprises and other organizations dramatically increases, more and more companies are looking to turn this data into actionable information and intelligence in real time. Addressing these requirements requires applications that are able to analyze potentially enormous volumes and varieties of continuous data streams to provide decision makers with critical information almost instantaneously. This IBM Redbooks® publication is written for decision-makers, consultants, IT architects, and IT professionals who will be implementing a solution with IBM InfoSphere Streams.

Typical deployment architectures introduce challenges to fully using the shared metadata platform across products, environments, and servers. Data privacy and information security requirements add even more levels of complexity. IBM® InfoSphere® Information Server provides a comprehensive, metadata-driven platform for delivering trusted information across heterogeneous systems. This IBM Redbooks® publication presents guidelines and criteria for the successful deployment of InfoSphere Information Server components in typical logical infrastructure topologies that use shared metadata capabilities of the platform, and support development lifecycle, data privacy, information security, high availability, and performance requirements. This book can help you evaluate information requirements to determine an appropriate deployment architecture, based on guidelines that are presented here, and that can fulfill specific use cases. It can also help you effectively use the functionality of your Information Server product modules and components to successfully achieve your business goals. This book is for IT architects, information management and integration specialists, and system administrators who are responsible for delivering the full suite of information integration capabilities of InfoSphere Information Server.

IBM® InfoSphere® Warehouse is the IBM flagship data warehouse platform for departmental data marts and enterprise data warehouses. It offers leading architecture, performance, backup, and recovery tools that help improve efficiency and reduce time to market through increased understanding of current data assets, while simplifying the daily operations of managing complex warehouse deployments. InfoSphere Warehouse Advanced Enterprise Edition delivers an enhanced set of database performance, management, and design tools. These tools assist companies in maintaining and increasing value from their warehouses, while helping to reduce the total cost of maintaining these complex environments. In this IBM Redbooks® publication we explain how you can build a business intelligence system with InfoSphere Warehouse Advanced Enterprise to manage and support daily business operations for an enterprise, to generate more income with lower cost. We describe the foundation of the business analytics, the Data Warehouse features and functions, and the solutions that can deliver immediate analytics solutions and help you drive better business outcomes. We show you how to use the advanced analytics of InfoSphere Warehouse Advanced Enterprise Edition and integrated tools for data modeling, mining, text analytics, and identifying and meeting the data latency requirements. We describe how the performance and storage optimization features can make building and managing a large data warehouse more affordable, and how they can help significantly reduce the cost of ownership. We also cover data lifecycle management and the key features of IBM Cognos® Business Intelligence. This book is intended for data warehouse professionals who are interested in gaining in-depth knowledge about the operational business intelligence solution for a data warehouse that the IBM InfoSphere Warehouse Advanced Enterprise Edition offers.

To make better informed business decisions, better serve clients, and increase operational efficiencies, you must be aware of changes to key data as they occur. In addition, you must enable the immediate delivery of this information to the people and processes that need to act upon it. This ability to sense and respond to data changes is fundamental to dynamic warehousing, master data management, and many other key initiatives. A major challenge in providing this type of environment is determining how to tie all the independent systems together and process the immense data flow requirements. IBM® InfoSphere® Change Data Capture (InfoSphere CDC) can respond to that challenge, providing programming-free data integration, and eliminating redundant data transfer, to minimize the impact on production systems. In this IBM Redbooks® publication, we show you examples of how InfoSphere CDC can be used to implement integrated systems, to keep those systems updated immediately as changes occur, and to use your existing infrastructure and scale up as your workload grows. InfoSphere CDC can also enhance your investment in other software, such as IBM DataStage® and IBM QualityStage®, IBM InfoSphere Warehouse, and IBM InfoSphere Master Data Management Server, enabling real-time and event-driven processes. Enable the integration of your critical data and make it immediately available as your business needs it.

Today, organizations face tremendous challenges with data explosion and information governance. InfoSphere™ Optim™ solutions solve the data growth problem at the source by managing the enterprise application data. The Optim Data Growth solutions are consistent, scalable solutions that include comprehensive capabilities for managing enterprise application data across applications, databases, operating systems, and hardware platforms. You can align the management of your enterprise application data with your business objectives to improve application service levels, lower costs, and mitigate risk. In this IBM® Redbooks® publication, we describe the IBM InfoSphere Optim Data Growth solutions and a methodology that provides implementation guidance from requirements analysis through deployment and administration planning. We also discuss various implementation topics including system architecture design, sizing, scalability, security, performance, and automation. This book is intended to provide various systems development professionals, Data Solution Architects, Data Administrators, Modelers, Data Analysts, Data Integrators, or anyone who has to analyze or integrate data structures, a broad understanding about IBM InfoSphere Optim Data Growth solutions. By being used in conjunction with the product manuals and online help, this book provides guidance about implementing an optimal solution for managing your enterprise application data.

What do you know about your data? And how do you know what you know about your data? Information governance initiatives address corporate concerns about the quality and reliability of information in planning and decision-making processes. Metadata management refers to the tools, processes, and environment that are provided so that organizations can reliably and easily share, locate, and retrieve information from these systems. Enterprise-wide information integration projects integrate data from these systems to one location to generate required reports and analysis. During this type of implementation process, metadata management must be provided along each step to ensure that the final reports and analysis are from the right data sources, are complete, and have quality. This IBM® Redbooks® publication introduces the information governance initiative and highlights the immediate needs for metadata management. It explains how IBM InfoSphere™ Information Server provides a single unified platform and a collection of product modules and components so that organizations can understand, cleanse, transform, and deliver trustworthy and context-rich information. It describes a typical implementation process. It explains how InfoSphere Information Server provides the functions that are required to implement such a solution and, more importantly, to achieve metadata management. This book is for business leaders and IT architects with an overview of metadata management in information integration solution space. It also provides key technical details that IT professionals can use in a solution planning, design, and implementation process.

In this IBM® Redbooks® publication, we discuss and describe the positioning, functions, capabilities, and advanced programming techniques for IBM InfoSphere™ Streams (V2), a new paradigm and key component of IBM Big Data platform. Data has traditionally been stored in files or databases, and then analyzed by queries and applications. With stream computing, analysis is performed moment by moment as the data is in motion. In fact, the data might never be stored (perhaps only the analytic results). The ability to analyze data in motion is called real-time analytic processing (RTAP). IBM InfoSphere Streams takes a fundamentally different approach to Big Data analytics and differentiates itself with its distributed runtime platform, programming model, and tools for developing and debugging analytic applications that have a high volume and variety of data types. Using in-memory techniques and analyzing record by record enables high velocity. Volume, variety and velocity are the key attributes of Big Data. The data streams that are consumable by IBM InfoSphere Streams can originate from sensors, cameras, news feeds, stock tickers, and a variety of other sources, including traditional databases. It provides an execution platform and services for applications that ingest, filter, analyze, and correlate potentially massive volumes of continuous data streams. This book is intended for professionals that require an understanding of how to process high volumes of streaming data or need information about how to implement systems to satisfy those requirements. See: http://www.redbooks.ibm.com/abstracts/sg247865.html for the IBM InfoSphere Streams (V1) release.

This IBM® Redbooks® publication will help you to better understand the effective use of the WebSphere® DataPower® family of appliances. It provides guidance on the best methods identified to date for building the various components that implement solutions, such as handling MQ-based message flows or creating authentication and authorization policies. The information and recommendations in this publication are the result of real world experiences using the appliances. Such experience shows that taking the time to plan a solution implementation before beginning the work yields the greatest savings in time and energy and the highest quality outcome. This publication begins with a checklist of items to consider when planning a DataPower solution. This publication is intended to provide answers or guidance to implementation problems often encountered by users of the appliance. This book is not intended to present complete solutions or templates because experience shows that every customer uses the appliance in their own unique environment with unique requirements. Thus, this publication provides a compendium of information about particular aspects of a solution. Use the Table of Contents or Index to find your current issue, and return to this publication when the next issue or question arises. Refer to the related IBM Redbooks publication entitled DataPower SOA Appliance Administration, Deployment, and Best Practices, SG24-7901 for more information.

In this IBM® Redbooks® publication, we present guidelines for the development of highly efficient and scalable information integration applications with InfoSphere™ DataStage® (DS) parallel jobs. InfoSphere DataStage is at the core of IBM Information Server, providing components that yield a high degree of freedom. For any particular problem there might be multiple solutions, which tend to be influenced by personal preferences, background, and previous experience. All too often, those solutions yield less than optimal, and non-scalable, implementations. This book includes a comprehensive detailed description of the components available, and descriptions on how to use them to obtain scalable and efficient solutions, for both batch and real-time scenarios. The advice provided in this document is the result of the combined proven experience from a number of expert practitioners in the field of high performance information integration, evolved over several years. This book is intended for IT architects, Information Management specialists, and Information Integration specialists responsible for delivering cost-effective IBM InfoSphere DataStage performance on all platforms.

IBM Information Server is a revolutionary new software platform that helps organizations derive more value from the complex heterogeneous information spread across their systems. It enables organizations to integrate disparate data and deliver trusted information wherever and whenever needed, in line and in context, to specific people, applications, and processes. IBM InfoSphere™ DataStage® is a critical component of the IBM Information Server, and the parallel framework of IBM InfoSphere DataStage is also the foundation for IBM InfoSphere QualityStage and IBM InfoSphere Information Analyzer components. This IBM® Redbooks® publication develops usage scenarios that describe the implementation of IBM InfoSphere DataStage flow and job design with special emphasis on the new features such as the distributed transaction stage (DTS) in Version 8.0.1, slowly changing dimensions stage ( Version 8.0.1), complex flat file stage (Version 8.0.1), and access to mainframe data.