Optimization problems get larger and more complex. Complex organizations need to plan their operations more effectively than ever before. A decision must be made. There are many ways to make it with existing scenarios and alternatives, as well as previously unknown options or approaches. All decisions are not equal. Some of the decisions are totally not acceptable due to some limits must be enforced. Furthermore, the decision must be compliance with business policies and regulations.

A decision-support environment is essential for business people to review their options and make confident decisions. Optimization technology has long been used to find the best solutions to complex planning and scheduling problems. It helps businesses make complex decisions and trade-offs about limited resources, better and faster.

Refer Mathematical Optimization for Faster and Better Decision-making.

Benefits of Optimization for Businesses

With a good optimization solution, decision makers can automatically evaluate millions of choices, gaining insights into input data, anticipating the best-case, expected-case and worst-case scenarios. We can understand trade-offs, alternatives, bottlenecks and inconsistencies, as well as sensitivities to various changes and relationships. Based on data, measurable targets allow the decision makers to rank alternative decisions and view results in new ways. Optimization assists us to take next best decisions by finding or developing the best possible plans or schedules that can be adapted in the course of operations. By streamlining the decision-making process, organizations can take actions that are more likely to achieve targets, also free up planners and operations managers so that they can leverage their expertise across a wider set of challenges. Big problems need sophisticated optimization capabilities.

Dilemma: Spreadsheet, Best-of-Breed, Custom & Platform

However, an optimization application that supports the flexible exploration of all the trade-offs and sensitivities is often missing. Besides open source which is yet a serious alternative, many organizations struggle with the dilemma of deciding between a commercially available, prepackaged, off-the-shelf, software application to address a particular decision or planning challenge versus creating something from scratch on their own. Here are the four questions that you can evaluate which is the best optimization option for your organization:

Is the financial opportunity strategically significant, i.e. will it make a difference in the value of the enterprise?
Do you need to capture as much of the opportunity as possible, i.e. you can’t settle for 80%?
Do you need to make these decisions to achieve this opportunity in a way that is different from the competition and that cannot be copied by them?
Do you intend to expand the use of this decision support technique and continuously improve the decision models you intend to build, thereby continuing to differentiate yourself in the market?

If the answer to any of the questions is “no”, then you should be looking at a package. If the answers to all four of these questions are “yes”, then you should be building a custom optimization application, not on off-the-shelf packages.

Why pre-built software applications are not the answer?

Spreadsheet

Spreadsheet-based software, like Excel is familiar and easy to use, suitable for small operation. However, it is not an enterprise class solution. It lacks of scalability because of the limited size of columns and rows. When the data is too much, it is very complex and hard to maintain, especially cumbersome to do the What-if analysis.

Best of breed

‘Best of breed’ is the packaged software with the out-of-the-box industry standard functionality and best practices. It takes less time to get started. However, it is a generic model software, everybody is having the same optimization capability, including your competitors. So, you have no competitive advantages. Since it is built for one size fits all, it may not capture all costs, constraints or goals of your specific business. It neither manage all business requirements, like GUIs, nor adapt to business changes.

Custom Applications

For a ever-growing long term business, the answer is always building a customized application that is tailored to your business needs from scratch. The issue with this option is always big investments in advanced in term of capital and human resources.

Platform

There is an increasingly viable fourth option – leveraging a platform for rapidly developing and supporting a secure, scalable, enterprise-class application that embeds powerful analytical techniques like optimization. The accelerating viability of platforms (in the cloud and otherwise) for developing your own applications has made this fourth option something that really can’t be overlooked.

You can have the benefits of both world, the pre-built software and custom software from scratch. It is easier for business managers to participate in model development process and dynamically adapats to changing business model with the rapid development and implementation cycle. Therefore, you have the unique competitive advantages over your competitors. It is lower development and maintenance cost but with full scalability and performance.

In most, if not all cases, this availability of a “platform” or “framework” means that doing something from scratch is now obsolete.

The entire Product Lifecycle Management (PLM) market is undergoing major changes caused by both customer business model changes and technology. It’s a microcosm of what we see with globalization and other market forces affecting clients’ paths to innovation. PLM is no longer viewed as just the software that engineers use to design products. PLM is now viewed by customers and analysts as the entire lifecycle of service and product creation, use S&S and retirement. The focus need to shift from a departmental or individual project view to a view of the role of entire value chain.

Consequently, the old definitions of software, services and consulting are morphing. The major driving force is customer business models. Efficiency and effectiveness in a global market drives greater need for product and service integration that is tightly coupled throughout the entire enterprise … and outside of it. All customers, big and small, exist in heterogeneous ecosystems which increasingly require data and information to flow freely, yet securely, to optimize leverage investments while being flexible for the future as well as cost efficient. This, in turn, helps foster innovation in places where we did not expect it before and opens up a world of opportunities for those who are partners and suppliers to client organizations.

Introduction to IBM’s Decision Optimization

IBM Decision Optimization is an integrated application development and deployment platform for building optimization models and solutions to tackle complex planning and constraint-based scheduling problems, in the most rapid and efficient way. It supports the entire process from model development to application generation to application refinement and data integration.

IBM Decision Optimization is an advanced tool for stochastic optimization. It automatically generates applications based on the structure of the underlying mathematical model. It will reduce reliance on PhD-level expertise as it improves the understanding of trade-offs (cost vs robustness) and increase model / solution stability and value. We can configure tabular and graphical views of data and solutions. The development team can engage the business users through early prototypes. Finally, embedding the optimization components in company-specific (external) applications without a hasle. The platform includes data analysis & visualization, scenario management & editing, collaborative planning and What-if / sensitivity analysis.

IBM’s market leadership and CPLEX’s performance history are powerful counter points to competitive comparisons.

The following diagram shows the deployment possibilities for applications developed with IBM Decision Optimization offerings.

Development IDE: IBM ILOG CPLEX Optimization Studio

CPLEX, or formally IBM ILOG CPLEX Optimization Studio is the heart product of the IBM Decision Optimization integrated system, is an Integrated Development Environment (IDE) for iterative model development. The IDE includes Editor, parser, profiler, solvers, DB connections and SPSS connector. CPLEX Studio is a good start for every optimization project. easy to access, preview available, self-service with model size restriction.

IDE offers full access to engines + much more development tools than competitors – increase development efficiency and speed. IDE uses the latest technology (Eclipse), IT compliant server technology state of the art CP tech, therefore smoother, better to connect to other technologies – advantage of integration / deployment. Better from usability standpoint – advantage of developer. Complete support of the development process: besides IDE + ML + 2 engines, also OPL, server, DOC. stable code (stability proven).

IBM ILOG CPLEX Optimization Studio provides powerful advanced analytics to transform descriptive data and predictive solutions into optimized prescriptive courses of action – replacing intuition and heuristic thinking with fact-based decisions. The users are the operational research experts, academia, OEMs, modeling experts and business decision makers with the helps of internal IBM partners (GSB, Products, … etc.). They use Optimization Programming Language (OPL), which is specifically designed to express optimization models.

For the detailed optimization model development, please read Optimization Model Development and Mathematical Techniques for Optimized Solution.

CPLEX (founded in USA, 1988) was the name of the company acquired by ILOG (founded in 1987) in 1997, then acquired by IBM in 2009. Besides CPLEX, other ILOG product lines, had been integrated into IBM. Sometimes, the CPLEX is also the optimization engine, CPLEX Optimizer.

Company background: IBM + ILOG = bigger R&D team than competitors
Engine + ML + IDE: 1 stop shopping is still attractive

Engine: IBM ILOG CPLEX Optimizer

We need an optimization engine with powerful mathematical processing capabilities to solve the model and find solutions. Read Optimization Engines Make The Industry Efficient for the history of optimization engines and how they make the industry efficient.

IBM ILOG CPLEX Optimizer is at the core component of the CPLEX Studio. It is a market leading optimization engine or solver with a set of search algorithms for mathematical programming and constraint-based scheduling, as well as for multiple programming and modeling interfaces. With IBM ILOG CPLEX Optimizer powerful algorithms, we can model business issues mathematically and solve them to produce precise and logical decisions. It can work unbundled from the CPLEX Studio by buying standalone deployment licenses for IBM ILOG CPLEX Optimizer.

IBM ILOG CPLEX Optimizer is a high-performance solver that tackle large real world problems with unmatched speed and reliability / stability, which had been proven in over a thousand mission-critical deployment. It can handle business problems with millions of constraints and decision variables.

We can embed the sophisticated optimization engines, i.e. CPLEX Optimizer, CP Optimizer and more, for creating smarter, resource-efficient plans and schedules.

CPLEX Optimizer automatically choose the most appropriate solution algorithm for a particular problem based on problem structure. Besides that, we can set the parameters and custom algorithm to have more control over the fundamental algorithms. The fundamental algorithms include linear, simplex, barrier, mixed integer, quadratic, constraint programming, quadratically constrained (including second-order cone programs) and mixed integer quadratically constrained programming optimizers, as detailed below:

Simplex Optimizer

Simplex Optimizer implements the primal simplex and dual simplex methods for linear and quadratic programming. It provides fast implementation of the network simplex method, which is specifically created for network problems, with or without side constraints.

Barrier Optimizer

Barrier Optimizer provides an alternative to the simplex method for solving linear and quadratic problems. It is an approach for solving quadratically constrained problems based on primal-dual predictor-corrector method. It includes Crossover algorithm, which converts the solutions created by the barrier algorithm into basic solutions that are typically provided by the simplex method and used for fast restarts and sensitivity analysis.

Mixed Integer Optimizer

Mixed Integer Optimizer solves mixed integer linear, mixed integer quadratic and mixed integer quadratically constrained problems. It employs a branch-and-cut technique that takes advantage of innovative strategies to provide high performance solutions for the hardest MIP programs. Included is preserve algorithm, sophisticated cutting-plane strategies and feasibility heuristics. We can customize cutting-plane and heuristics strategies, include their own when problem-specific techniques are valuable.

Mixed Integer Optimizer provides default and parameter settings that work well for most problems. We can customize the search strategy or select specialized techniques. We can state whether it is more important to find an optimal solution or quickly determine a good feasible solution, Optimizer will automatically adjust solution strategies to provide the desired behavior. Finally, we can direct the optimizers to return multiple solutions to explore the effects of subjective preferences on the solution space.

Parallel Optimizers

Parallel Optimizers automatically take advantages of multiple CPUs to solve extremely difficult problems with CPLEX Parallel Barrier and CPLEX Parallel MIP Optimizers. CPLEX Parallel Barrier Optimizer achieve significant speedups over its serial counterpart on a wide variety of classes of linear programming problems. The CPLEX Parallel MIP Optimizer implements a deterministic mode of operation that produces a repeatable, invariant solution path and opportunistic mode that takes full advantage of parallelism for potentially better performance.

Parallel Optimizers can substantially reduce the time for solving large linear and difficult MIP programs. In an alternative use of multiple processors, CPLEX Concurrent Optimizer uses different algorithms on different CPUs in a race to fond the best way to solve a problem.

CP Optimizer (CPO)

Besides that, CPLEX consists of CPO. CP Optimizer automatically computes optimized detailed schedules from a constraint programming model. A scheduling model is expressed in a declarative fashion, using decision variables, constraints and objectives that must be minimized or maximized.

CPO is designed to excel at computing the optimal sequence of large numbers of inter-related activities. Created for case of use, it is a natural fit for those familiar with traditional modeling techniques for MIP.

Programming interface

CPLEX Optimizer can be easily accessible from multiple interfaces. It is compatible with mathematical programming tools from AIMMS, AMPL, GAMS, MPL and Microsoft Solver Foundation. There are connectors to Microsoft Excel and Mathlab. CPLEX Optimizer provides both a traditional matrix orientation for representing mathematical models and a higher-level representation that uses modeling objects to help the developer to take advantage of object-oriented programming approaches. Developers can state their constraints in the traditional mathematical programming form, using linear or quadratic expressions on these modeling objects or alternatively they can state constraints using logical expressions to improve readability and maintainability.

Component libraries provides the features and flexibility that developers need to create customized applications for solving both simple and complex optimization problems. The libraries include C, C++, .NET, Java and Python programming interfaces that help developers to efficiently embed CPLEX Optimizer in our software applications, while other languages are supported through the C interface. The programming interfaces provide a comprehensive set of routines for defining, solving analyzing, querying and creating reports for optimization problems and solutions. Modeling objects are easily accessible to programmers using C++, Java, .NET and Python languages.

Interactive Optimizer

Interactive Optimizer is a command-line utility that lets users read and write mathematical programming problem files and tune the performance of CPLEX Optimizer to the requirements of a specific problem.

Deployment Platform: IBM ILOG Optimization Decision Manager (ODM) Enterprise

IBM ILOG ODM Enterprise (ODME) enables easy and rapid development and deployment of flexible optimization technology-based analytical custom decision support applications in all aspects.

Balancing conflicting goals

In the real world, multiple business objectives (such as minimizing various costs, increasing customer service, revenue and profitability) often conflict. ODME helps balancing / adjusting these conflicting business assumptions, objectives and goals, enabling informed trade-offs and helping users to change the weights associated with each goal or set individual goals, meanwhile judging the impact of data changes. In practice, working with goals is as important as working with costs and operating constraints.

Relaxation of constraints

In a ODME-based application, over-constrained problems are automatically relaxed during runtime by IBM ILOG CPLEX, which is careful to relax the fewest and least important constraints. Controlled relaxation of binding constraints (by group) ensures that a solution is always found and presented along with the information about relaxed preferences or constraints.

Automatic application generation

ODME slash development costs and saves time by automatically generating a complete decision-support application from the structure of an OPL model that associated with scenario creation and comparison, controlled relaxation of constraints and goal balancing.

Optimization Programming Language (OPL) is a specialized optimization language / syntax that helps professionals build models faster with fewer lines of code with scripting capability. It is an IBM transparent modeling language, to specify the targets, limits and choices related to your business.

A simple wizard-guided step provides the initial application. OPL data structures are automatically mapped to data tables in IBM ILOG ODM Enterprise, decision variables and solution metrics are mapped to solution views and objective functions are mapped to IBM ILOG ODM Enterprise’s interactive business goals. Constraints marked as relaxable in the OPL model are exposed to the end user by name and priority, allowing relaxations and trade-offs based on business importance. The application generation is also available as CPLEX on z/OS and CPLEX with CP Optimizer on zLinux.

Visual of Complex datasets and solutions

Charts based on IBM ILOG Visualization products are pre-packaged with IBM ILOG ODM Enterprise for creating a variety of tabular and graphical views of input data and solutions. Data analysis with drill-down and graphical display, enabling detailed visualization of data and decisions, allowing developers to easily produce informative displays for solution analysis and scenario comparison. Furthermore, business professionals can visualize the engine’s optimal solutions or recommendations in graphical interface and familiar business terminology explanations.

Data and Custom GUIs Configuration

To further configure and tailor the generated application, the IBM ILOG CPLEX Optimization Studio contains editors for customizing the appearance and behavior of all data display tables and charts, business goals, business costs and business constraints in an IBM ILOG ODM Enterprise application. In addition, any type of custom Java-based graphics can be created and embedded into the IBM ILOG ODM Enterprise GUI, allowing final applications to include completely custom views. This is also available for LoB through ODM Enterprise Client or Planner Edition.

Embedding through API’s

By means of application programming interfaces (APIs), models created by IBM ILOG CPLEX Optimization Studio can be easily integrated into external applications written in Java, C++ or .NET, without having to rewrite the model in other programming languages. This capability enables you to include optimization components developed with the CPLEX Optimization Studio in your own corporate IT applications, taking advantage of existing business process models, databases and user interfaces. This also enables tight integration with other business systems.

CPLEX Enterprise Server (CES)

IBM ILOG ODM Enterprise is focused on producing scalable enterprise deployable, state-of-the-art decision support applications. More than that, at any point in the model development process, a draft model can be turned into a working application. You can implement your CPLEX optimization solution as a client application in Java, by deploying the OPL project to CPLEX Enterprise Server. You can store it in a database and extract it on the server to be used by the client interaction with the ongoing optimization process.

Collaborative planning

Optimization model development is about 75% management consulting and only 25% optimization expertise, involving economic analysis, cost accounting, business strategy and process modelling. To carefully develop good optimization models, the busines must be analyzed and many people must be interviewed. Many views of a problem are then assembled and structured. The large amount of historic and real-time data must be analyzed to obtain correct costs and yields. We need to define the exact decisions that need to be made and the metrics that should be part of the model. The iterative model development process is with the full participation of the necessary range of stakeholders from multiple functions, i.e. operational managers, financial analysts and business planners.

Through a central repository, planners can collaborate in the process of refining models, model behavior, displays and scenario creation parameters. The collaborative planning system / workflow defines the business user roles and users access rights in business process support. This makes it easier for remote planners and business management to participate and support large-scale applications in distributed planning processes. Business analysts and managers can easily understand inputs and outputs, and can ask questions and make suggestions.

With IBM, users also have the ability to find and work with experienced people who have built custom planning and scheduling applications before:

facilitating the cross-functional model definition and refinement process
building, testing, tuning, and debugging the model
data preparation, manipulation and testing
mapping optimization data arrays to tables

Decision Optimization Center (DOC)

DOC is a solution platform for optimized decision, planning and scheduling solutions. The enabler for Business Value solutions for:

IT Architect
OEM / ISV
Consultant
Business partners
Internal IBM partners

DOC includes:

CPLEX Studio development right to develop optimization models.
Optimization server, besides CES.
Out of the box GUI for optimization-based application creation.
JViews development right for customizing GUI.
WebSphere Application Server (WAS) for development.

Predictive Analytics meets Optimization to Automate Decision Making

In our previous post, Mathematical Optimization for Faster and Better Decision-Making, we discussed about the Advanced Analytics Landscape, 3 levels of analytics sophistication.

Descriptive Analytics: IBM Cognos

The descriptive analytics is the simplest sophistication of analytics. It allows us to condense big data into smaller, more useful summary of information. With the enough data in hand, we start to see patterns and relate the data in a way which may shed light on connections.

IBM Cognos helps business users who needs a proven, scalable analytics solution to easily create compelling visualizations and recomended dashboards in diverse formats. It reduces the time needed for data preparation through automation and built-in intelligence.

Predictive Analytics: IBM SPSS

Predictive analytics is the next step up in data reduction. It utilizes a variety of statistical methods, modeling, data mining and machine learning techniques to study the patterns of the recent and historical data. These include preparing and managing the data through scoring and classification. We need tools that help us to use statistical methods for predicting or forecasting. It can give us prediction what will happen if things go unchanged.

The IBM SPSS software platform offers advanced statistical analysis, a vast library of machine-learning algorithms, text analysis, open-source extensibility, integration with big data and seamless deployment into applications. The IBM SPSS can:

help organizations to predict with confidence what will happen next so that they can make smarter decisions and improve business outcomes.
transform data into predictive insights to guide frontline decisions and interactions.
predict what customers want and will do next to increase profitability and retention.
maximize the productivity of people, processes and assets.
measure the social media impact of products, services and marketing campaigns.

Prescriptive Analytics: IBM SPSS Modeler Gold + IBM Decision Optimization

The prescriptive analytics is an extension of predictive analytics with two additional components, i.e. actionable data and a feedback system that tracks the outcome produced by the action taken.

The SPSS Connector in CPLEX Studio can take SPSS Modeler stream as input data to execute an optimization model. This synergy provides SPSS and CPLEX users a powerful end to end decision support system to automate, predict, optimize and govern repeatable complex business decisions. It fuels the optimization formulas of the packaged templates and widens possible use cases that can be addressed.

Cloud Capacity Optimization Example

Cloud Big Data

Collect Static Data
- Host profiles
- VM Profiles
- Host-VM Mapping
Collect Dynamic Data
- VM CPU Usage
- VM Memory Usage
- VM Network Usage
- VM Disk Usage

Descriptive Analytics

Dashboard showing real-time analysis results and actions
Real-time VM moves or re-assignments
Real-time host activation / de-activation
Real-time Cloud data (e.g. cpu / memory usage)

Predictive Analytics

Predict Cloud activity
- VM CPU Usage
- VM Memory Usage
Use predictive models (e.g. times series models) built using IBM SPSS Modeler integrated with IBM InfoSphere Streams.

Prescriptive Analytics

Optimize the cloud configuration in real-time by re-assigning VMs to hosts.
Minimize the number of physical hosts (minimize cost & energy).
Turn off unused hosts.
Move VMs to different hosts dynamically.
Minimizing the number of VM moves / migrations.
Uses IBM ILOG CPLEX InfoSphere Streams.

If you are ready to leverage IBM Decision Optimization to develop and deploy an optimization application customized to your organization, please read Optimization Licensing & Configuration for IBM ILOG CPLEX & Decision Optimization Center for cost estimation details.