Comment to BBBT Blog on Wherescape

Today started for me with a great Bouder BI Brain Trust [BBBT] Session featuring WhereScape and their launch of WhereScape 3D [registration or account required to download], their new data warehouse planning tool. Other than my interest in all things related to data management and analysis [DMA], the WhereScape 3D tool is particularly interesting to me in its potential for use in Agile environments and its flexibility in being used with other data integration tools, not just WhereScape Red. Richard Hackathorn does a great job describing WhereScape 3D, which launched in beta at the BBBT, complete with cake for those in the room, which he's already downloaded and used. [I'm awaiting the promised cross-platform JAR to try it out on my MacBookPro.]

Unfortunately, Twitter search is letting me down today as I normally gather all the #BBBT tweets from a session, send them to Evernote, and check these "notes" as I write a blog post.

WhereScape 3D is a planning tool, allowing a data warehouse developer to profile source systems, model the data warehouse or data mart, and automagically create metadata driven documentation. Further, one can iterate through this process, creating new versions of the models and documentation, without destroying the old. The documentation can be exported as HTML and included in any web-based collaboration platform. So, there is the potential of using the documentation against Scrum style burn down lists and for lightweight Agile artifacts.

WhereScape 3D and Red come with a variety of ODBC drivers, and, with the proper Teradata licensing, the Teradata JDBC driver as well. One can also add other ODBC and JDBC drivers. However, neither WhereScape product currently allows connections to non-relational database sources. I would find this to be severely limiting, as in traditional enterprises, we've never worked on a DMA project that didn't include legacy systems requiring us to pull from flat files, systems written in Pick Basic against a UniVerse or other multi-value database management system [MVDBMS], electronic data interchange [EDI] files, XML, or java or ReSTful services. In other cases, we're facing new data science challenges of extreme volumetric flows of data from web, sensor and transaction logs, requiring real-time analytics, such as can be had with SQLstream, or stored in NoSQL data sources, such as Hadoop and its offshoots.

Which leads us to another interesting feature of WhereScape 3D: it's designed to be used with any data integration tool, not just WhereScape Red. I'm looking forward to get that JAR file, currently hiding in a MS Windows EXE file, and trying WhereScape 3D in conjunction with Pentaho Data Integration [PDI or KETTLE] and seeing how the nimble nature of WhereScape 3D planning works with PDI Spoon AgileBI against all sorts of data flows targeting LucidDB ADBMS and data vault. Yeehah!

Full360 on BBBT

Today, Friday the 13th of May, 2011, the Boulder BI Brain Trust heard from Larry Hill [find @lkhill1 onTwitter] and Rohit Amarnath [find @ramarnat on Twitter] of Full360 [find @full360 on Twitter] about the company's elasticBI™ offering.

Serving up business intelligence in the Cloud has gone through the general hype cycles of all other software applications, from early application service providers (ASP), through the software as a service (SaaS) pitches to the current Cloud hype, including infrastructure and platform as a service (IaaS and PaaS). All the early efforts have failed. To my mind, there have been three reasons for these failures.

  1. Security concerns on the part of customers
  2. Logistics difficulties in bringing large amounts of data into the cloud
  3. Operational problems in scaling single-tenant instances of the BI stack to large number of customers

Full360, a 15-year-old system integrator & consultancy, with a clientele ranging from startups to the top ten global financial institutions, has come up with a compelling Cloud BI story in elasticBI™, using a combination of open source and proprietary software to build a full BI stack from ETL [Talend OpenStudio as available through Jaspersoft] to the data mart/warehouse [Vertica] to BI reporting, dashboards and data mining [Jaspersoft partnered with Revolution Analytics], all available through Amazon Web Services (AWS). Full360 is building upon their success as Jaspersoft's primary cloud partner, and their involvement in the Rightscale Cloud Management stack, which was a 2010 winner of the SIIA CODiE award, with essentially the same stack as elasticBI.

Full360 has an excellent price point for medium size businesses, or departments within larger organizations. Initial deployment, covering set-up, engineering time and the first month's subscription, comes to less than a proof of concept might cost for a single piece of their stack. The entry level monthly subscription extended out for one year, is far less than an annual subscription or licensing costs for similar software, considering depreciation on the hardware, and the cost of personnel to maintain the system, especially considering that the monthly fee includes operations management and a small amount of consulting time, this is a great deal for medium size businesses.

The stack being offered is full-featured. Jaspersoft has, arguably, the best open source reporting tool available. Talend Open Studio is a very competitive data integration tool, with options for master data management, data quality and even an enterprise service bus for complete data integration from internal and external data sources and web services. Vertica is a very robust and high-performance column-store Analytic Database Management System (ADBMS) with "big data" capabilities that was recently purchased by HP.

All of this is wonderful, but none of it is really new, nor a differentiator from the failed BI services of the past, nor the on-going competition today. Where Full360 may win however, is in how they answer the three challenges that caused the failure of those past efforts.


Full360's elasticBI™ handles the security question with the answer that they're using AWS security. More importantly, they recognized the security concerns as one of their presentation sections today stated, "Hurdles for Cloud BI" being cloud security, data security and application security. All three of these being handled by AWS standard security practices. Whether or not this is suficient, especially in the eyes of customers, is uncertain.


Operations and maintenance is one area where Full360 is taking great advantage of the evolution of current Cloud services best known methods and "devops" by using Chef opscode recipes for handling deployment, maintenance, ELT and upgrades. However, whether or not this level of automation will be sufficient to counter the lack of a multi-tenant architecture remains to be seen. There are those that argue that true Cloud or even the older SaaS differentiators and ability to scale profitably at their price-points, depends on multi-tenancy, which causes all customers to be at the same version of the stack. The heart of providing multi-tenancy is in the database, and this is the point where most SaaS vendors, other than salesforce-dot-com (SFDC), fail. However, Jaspersoft does claim support for multi-tenant architecture. It may be that Full360 will be able to maintain the balance between security/privacy and scalability with their use of devops, and without creating a new multi-tenant architecture.Also, the point of Cloud services isn't the cloud at all. That is, the fact that the hardware, software, platform, what-have-you is in a remote or distributed data center isn't the point. The point is the elastic self-provisioning. The ability of the customer to add resources on their own, and being charged accordingly.

Data Volume

The entry-level data volume for elacticBI™ is the size of a departmental data mart today. But even today, successfully loading into the Cloud, that much data in a nightly ETL run, simply isn't feasible. Full360 is leveraging Aspera's technology for high-speed data transfer, and AWS does support a form of good ol' fashioned "sneaker net", allowing customers to mail in hard drives. In addition, current customers with larger data volumes, are drawing that data from the cloud, with the source being in AWS already, or from SFDC. This is a problem that will continue to be an "arms race" into the future, with data volumes, source location and bandwidth being in a three-way pile-up.

In conclusion, Full360 has developed an excellent BI Service to suplement their professional services offereings. Larger organizations are still wary of allowing their data out of their control, or may be afraid of the target web services provide for hackers, as exemplified by the recent bank & retailer email scammers, er marketing, and Sony break-ins. Smaller companies, which might find the price attractive enough to offset security concerns, haven't seen the need for BI. So, the question remains as to whether or not the market is interestd in BI in the Cloud.

This post was simultaneously published on the Blog of the Boulder BI Brain Trust, of which I'm a member.

Welcome to the Data Archon

We've been blogging for over five years on data management & analytics open source solutions, collaboration, mobile, project management, Agile implementations and the like. In addition to these topics, we'll be discussing statistical, mathematical and computerized modeling, management and analysis of data.

Agile Implementation of Data Management and Analytics

Agility in Data Management and Analytics

Our 8D™ Methodology was declared to be in compliance with the Agile Manifesto in 2002 by an Agile Practitioner, Todd McGrath of supergloo, inc. There are three parallel tracks, with the eight dimensions of the project iterating across the Strategic, Tactical and Implementation Tracks throughout the entire life of the data management and analytics (DMA) program. For example, test scenarios are written as part of the strategic track, refined into test plans as part of the tactical track, and the test scripts are written against the scenarios and plans in each implementation sprint. As another example, this methodology is all about data; the data distribution, simpler business process and data models, and most importantly, what questions we wish to ask of the data, and what stories the data can support.

The Strategic Track concentrates on modeling business processes and enterprise data. We use our Analysis by Interview™ to respond to changes in user needs and the business environment. The strategy is not static. As the parallel Implementation track is executed, it iteratively updates the strategy, architecture and models that are done in the Strategic track. Projects are managed and scoped so that each project is designed to be completed in 90-days, with a focused set of deliverables. Project Management Techniques ensure manageability and success of both small projects, and large projects with multiple 90-days implementation phases.

Our major emphasis is on working with the end users, development personnel and operations staff to assure that all parties are involved and informed, and that their needs are heard. Years of experience have evolved into specific techniques for integrating all concerned individuals into a project team, garnering their ideas and concerns, and disseminating project information. These techniques are codified as our Analysis by Interview™ method. The biggest advantage of this methodology is that we are able to analyze, prioritize and synthesize user requirements to ensure that the systems developed provide the greatest value and truly reflects the needs of the organization.

Information, needs and concerns are generally gathered and recorded in blogs. Lightweight artifacts are created and maintained as wiki pages. Meetings are documented and shared through audio and video recordings, as well as notes in the wiki.

The Tactical Track focuses on solving a specific business need, product package or decision support goal as part of bringing the Strategic Track to fruition.

The Implementation Track consists of sprints that bring specific functions and capabilities into production in support of a Tactical Track objective.

D1- Direct [the DMA program]: The Direct Stage kicks-off the DMA program by laying out the framework for the Strategic, Tactical and Implementation Tracks. During this stage, the project team(s), indirectly involved stakeholders,

D2 - Discover [the problem or need]: The objective of the Discover Stage is to produce, with user involvement, a set of business models, a set of recommendations and an agreed upon plan for the development process, which will serve the organization’s current and future needs. It takes into account the organizational, financial and technical constraints. It will also provide a stable framework that can be used to focus the work of several tactical phases, as they each progress through the stages of analysis, design, development, user documentation, transition and production.

D3 - Determine [the solution]: The Determine Stage performs detailed analysis for the specific areas of the organization recommended in the Discover Stage. It verifies all the findings from the discover stage and expand these into sufficient detail to ensure accuracy of business functions. It also ensures the solid foundation for design that bears in mind business processes of the organization and its existing systems.
This stage of the project consists of the following activities:

  1. Determine the architectural strategy
  2. Map business requirements to standard process flows and application functions
  3. Conduct cooperative application planning sessions
  4. Define backup, business continuity and security strategies
  5. Determine required training for users and support personnel
  6. Finalize project plan and functional requirements

D4 - Design [the specifications]: The Design Stage has a very straightforward and clear-cut objective: to have a detailed framework of standards and specifications, reflecting and iterating the strategic and tactical set, in place to assure successful construction and development of the sprint deliverables. There are parallel streams of activities during this stage, including:

  1. Formal and final determination of the business functionality that will be delivered
  2. Data analysis, data mapping, modeling and design specific to each objective
  3. Systems and Infrastructure research
  4. Tools/Reports/Analyses/Dashboards Evaluation and Selection

Some deliverables of this stage include: data models, detailed project plan for the Development Phase, Updated Risk Assessment, Design Document and Specifications and Design Review Report.

D5 - Develop [the modules]: The Develop Stage will code and test software. This stage involves planning, design of program structure, coding, bottom-up testing (Unit and Link Tests), top-down testing (System Test) and a disciplined approach to doing the work and controlling the versions of the scripts, programs, test packs, and user interface.
Some key success factors during this stage include:

  1. Ensuring quality work in tight time schedules
  2. Identification of early indications of performance, e.g. network, input/output or CPU bottlenecks
  3. Continuous tuning of the programs and the database
  4. Versioning/Release Control, Configuration Management, Change Control
  5. Software Quality Assurance
  6. Testing Limits and Exceptions

A test plan will be created to provide procedures and standards for the project testing activities. The test plan will include test cases, test schedules and test results guidelines.
Testing will include unit, integration and system tests as well as user acceptance testing. Test results are documented and feed back on system quality (technical and functional) and performance are used to improve the system.

D6 - Deploy [the system]: The Deploy Stage performs all the tasks necessary for implementation and provides an initial period of support for the system. The deploy stage is accomplished without disrupting the organization, and leaves the users confident and open to exploit the developed system. During this stage formal training for both business and technical staff is provided to supplement the on-going technology transfer of the previous stages. The user community is assisted in performing an acceptance test. Some activities and key deliverables are:

  1. Training and documentation
  2. Converted data
  3. Implemented application(s)
  4. Post-Implementation review report
  5. Train business users and technical support personnel
  6. Perform system cut-over
  7. Provide any necessary problem resolution

One important consideration that applies to every project is the maintenance and evolution of the final product. This process is dedicated to knowledge transfer to our customer personnel as well as within our team.

D7 - Decide [the next steps]: The Decide Stage consist of assessing the delivered implementation, conducting “lessons learned” sessions with all participants and providing recommendations on the next steps and future iteration and enhancements. It must be noted that a data warehouse, as in any system, evolves with the organization. Its functionality must continuously reflect the needs of the user community. The 8D™ Methodology is applied to each tactical and implementation step performed in the project. The results of the Decide Stage iteratively feed the lessons learned in the implementation to the Strategic Track. Thus the project strategy is gradually updated to assure that lessons learned and changing user expectations are met by the project.

D8 - Document [through living, lightweight, online processes]: The Document Stage proceeds throughout the program, using blogs, wiki, audio and video recordings, and other methods to assure that the "as desired", "as planned" and "as built" implementations are recorded and reconciled throughout the life of each tactical project and the overall strategic program.

The 8D™ Methodology successfully addresses the complex nature of the system life cycle through tight interaction between the strategic, tactical and implementation tracks of the program, projects and sprints. Project Management, documentation, user involvement, testing and technology transfer are the components that make it effective.

Moved from OSS to Data blog in 2010

May 2019
Mon Tue Wed Thu Fri Sat Sun
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    
 << <   > >>
We take a system and ecosystem approach to data management and analytics, with a focus on developing Sensor Analytics Ecosystems for the Internet of Things. As Independent Researchers we work with data management and analytics vendors to understand the aspects of IoT data and metadata such as time-series, location, sensor specifications & degradation; we work with IoT vendors to understand their data management and sensor analytics needs; we work with both for adaption to Sensor-Actuator Feedback Loops interacting through the Fog, Edge, Intermediate Aggregation Points, Cloud and Core, with augmenting decisions at every point, and making autonomous decisions as IoT mature through the 5Cs: Connection, Communication, Collaboration, Contextualization and Cognition. We work with Academics, Technology-for-Good, Government and Business Organizations to understand advances in Science, Technology, Engineering, Arti and Mathematics. We filter this information through a framework of Cultural, Regulatory, Economic, Political and Environmental factors to imagine future scenarios that allow our customers to gauge adoption without the hype. We work individually and with partners to develop strategies, define system and enterprise architectures, manage programs and projects, and achieve success with IoT. 37.652951177164 -122.490877706959


  XML Feeds