Open Source Solutions

Just got home from the MySQL Data Warehousing and BI Birds of a Feather gathering (BoaF). I'm tired, but my mind is on overdrive. 'Tis a great feeling.

First, I want to thank Lance Walter of Pentaho for introducing Clarise and me to the group as publishers of the OSBI Lens on Squidoo, this blog and the OSS wiki.

Clarise and I had a great conversation with Dr. Jacob Nikom of the MIT Lincoln Laboratory. The conversation ranged from the great Chimay beer that Matt Casters of Pentaho and Lead Architect of KETTLE, brought with him from Belgium, to

• Data Modeling and Relational Algebra/Theory: the integrity of the model
• Bayesian Statistics, Weibull Analysis and Tensor Calculus for mathematical modeling of complex systems [I love it when n-dimensional eigenvalues start floating in front of my eyes]
• Meeting the needs of different types of users: managers, scientists, business folk
• supplementing historical data warehouses with [near] real time data using ESB and dashboards
• data visualization of complex data sets such that the analyses and limitations can be grasped at different levels by different users
• collaboration among distributed workgroups of disparate career backgrounds and cultural pre-dispositions
• use of Second Life and other virtual worlds for collaboration and data visualization
• a calculator is to a computer [think if statment] as a flat file is to a relational database [think where clause]
• early USSR vs. British knock-offs of IBM mainframes
• Complexity as the balance of robustness and fragility

At various times in this discussion, we were joined by Sherman Wood, Director of BI at JasperSoft, and one of the legendary Mondrian developers, Julian Hyde of Pentaho and Mondrian Lead Architect, and Nicholas Goodman Director of BI for Pentaho.

And if you put Nick and dashboards and virtual worlds in the same post, then you have to mention Discoverer meets Duke Nukem.

Jakob, et al, thank you so very much for a great conversation.

Here's a guide to what's happening in open source business intelligence and data warehousing at the 2007 MySQL Conference and Exposition.

Speakers

• Fabrice Bonan is co-founder and Chief Operating Officer of Talend.
• Roland Bouman has been awarded the MySQL 2005 Beta Challenge, and the MySQL 2005 Forum Leader of the year.
• Matt Casters is Pentaho's Chief Architect for Data Integration and the founder of the KETTLE open source data integration tool.
• James Dixon is Pentaho's CTO and Chief Geek.
• Virgil Dodson is a Technology Evangelist at Actuate Corporation.
• Victoria Eastwood is a consultant with Infobright Inc.
• Ian Fyfe is currently director of product management and product marketing at JasperSoft Corporation.
• Julian Hyde is Pentaho's Chief Architect for OLAP, and the founder and lead for the Mondrian OLAP project.
• Barry Klawans is responsible for the technical vision and direction of JasperSoft Corporation's products.
• Brian Miezejewski is a Principal consultant for MySQL.
• Lance Walter is Pentaho's VP of marketing.

Sessions

• Leveraging Open Source for Operational Data Integration
• BI4DBAs: Using Pentaho, BIRT, and Eclipse for Monitoring MySQL
• Addressing Data Chaos: Using MySQL and Kettle to Deliver World-class Data Warehouses
• Exploiting MySQL 5.1 for Advanced Business Intelligence Applications
• Web-based Ad-hoc Query and Reporting with Pentaho
• Flexible End-user Reporting from MySQL with BIRT
• Data Compression and Analytic Query for Data Warehousing
• Using the JasperSoft BI Suite for Data Integration, Reporting, and Analysis with MySQL
• Building Scalable OLAP Applications with Mondrian and MySQL
• Embedded JasperSoft for MySQL Inside Enterasys
• Integrating MySQL and JasperReports
• High Performance Data Warehousing with MySQL: Tricks and Tips from the Field

Exhibit Hall

The Exhibit Hall Hours are:

• Tuesday, April 24 - 10:00 a.m. - 4:30 p.m.
• Tuesday Night Reception - 6:00 p.m. - 7:30 p.m.
• Wednesday, April 25 - 10:00 a.m. - 5:00 p.m.

OSBI related companies with booths at MySQL 2007 include Actuate Corporation, JasperSoft in Booth #409, Pentaho, and Talend in Booth #606.

And don't miss the Birds of a Feather gathering (BoaF): MySQL Data Warehousing and BI with Roland Bouman, Matt Casters, James Dixon, Julian Hyde, Brian Miezejewski, Lance Walter, and Nicholas Goodman. The related sessions on GIS, scalability, performance and tuning, clusters, and The Declarative Power of Views, all look very interesting.

See you there.

We discovered a new OLAP engine, Cubulus, registered on Sourceforge on 2007-03-07. The developer has an interesting presentation on the web, as well. [Just click anywhere on the page that isn't a link to somewhere else, and you'll go to the next slide. Or was that obvious to everyone but me?] Even though the project is pre-alpha, we've added it to our linkblog [look in the side column under OLAP]. The first database supported is MySQL, but they've recently added "hacks" for PostgreSQL support as well.

JasperSoft will have a booth and be presenting on three topics at the MySQL Conference & Expo 2007.

2007 MySQL Conference & Expo
April 23 - 25, 2007
Santa Clara Convention Center
Santa Clara, California
Booth #409 at the Exhibitors' Hall

2007 MySQL Conference & Expo
April 24, 2007
3:35 PM - 4:35 PM
Santa Clara Convention Center
Santa Clara, California
Using the JasperSoft BI Suite for Data Integration, Reporting, and Analysis with MySQL / Barry Klawans

2007 MySQL Conference & Expo
April 25, 2007
1:40 PM - 2:25 PM
Santa Clara Convention Center
Santa Clara, California
Integrating MySQL and JasperReports / Barry Klawans

2007 MySQL Conference & Expo
April 26, 2007
2:30 PM - 3:15 PM
Santa Clara Convention Center
Santa Clara, California
Embedded JasperSoft for MySQL Inside Enterasys / Barry Klawans

In addition, JasperSoft is a Platinum sponsor of this year's conference. For other upcoming events in which JasperSoft is a participant, visit their events page.

As we cite in our OSBI Daily, an integrated reporting solution is coming to OpenOffice.org as reported by Pentaho and Sun Microsystems today, planned to be incorporated into the next major release of OpenOffice.ORG, version 2.3.

The jFreeReport open source report project was brought into Pentaho in late 2005, and has been enhanced with "drag-and-drop report designer, MDX support, and integration with the Pentaho platform for report scheduling, security, and portal integration[;] Pentaho's recently-announced standards-based metadata layer and AJAX-based ad hoc query tool will also support and extend Pentaho Reporting's capabilities" allowing OpenOffice.org users "to create reports with content from the OpenOffice.org Base database as well as a wide range of proprietary and open source relational databases, OLAP and XML sources".
and read more on GullFOSS

I see this as a very powerful development for Open Source adoption. This shows the power of

1. the open document format (ODF) and open standards in general
2. interoperability among open source projects and solutions, and
3. alliances among open source companies and supporters.

This is one more example of how open source business intelligence solutions and open source desktop applications can give established proprietary vendors a run for the money. While many open source commercial organizations aren't yet comfortable with competing head-to-head with large enterprise vendors and like to use the mantra "80% of the capabilities at 20% of the cost", I think that the open source platforms, appliances and solutions coming from the likes of JasperSoft, Pentaho, SpagoBI, Greenplum, Ingres, Mulesource, Alfresco and Sun Microsystems prove that open source can compete for large-enterprise, government, university and SMB customers with their combination of the advantages of community, flexibility and standardization, powerful and extensive functions, and amazing cost effectiveness.

Of course, being Apple users, we have to hope that this solution will quickly find its way into NeoOffice, once it's proven out in OpenOffice.org 2.3.

Welcome to the wonderful world of OSBI [not Oklahoma State Bureau of Investigation] Our linkblog now has over 60 links to OSBI projects, and other sites. I hope that you find it useful. We'll be updating our wiki and lens on Squidoo to match it.

I just ran across the BI Team Blog from Face and Hannibal, as they "uncover open source business intelligence", discovered via Technorati. I tried to leave a comment, but I'm having this little problem.

At any rate, I've added the BI Team to our linkblog. I plan to follow along as they check out Pentaho and JasperSoft BI Solutions. Check them out.

Upadate 20070415;13h05: I had set this post up as a trackback to the BI Team, and had recieved an "OK" back from the wordpress.com server, but I see it hasn't appeared 17 hours later. /sigh Let's try again... Hmm, wordpress.com says that they already have a ping from us for that post.

Yesterday I spent the afternoon with Andrew Lampitt, Senior Manager Business Development, Nick Halsey, VP of Marketing, and Ian Frey, Director of Product Management & Product Marketing, at JasperSoft HQ in downtown San Francisco.

We discussed a wide variety of topics such as all of our backgrounds, JasperSoft's history, strategies, and future plans. Here's a sampling of the conversation.

• As announced in January, JasperETL is based upon Talend Open Studio. JasperSoft found Talend Open Studio to be a very mature and well-planned data integration product. While the Talend Open Studio ETL tool was only released in the latter part of 2006, it is the result of a three year R & D effort led by former Informatica and DataStage personnel system integrators [updated 20070416]. The goal of JasperETL is to provide an easy-to-use but fully featured graphical ETL tool to facilitate data integration for the Jasper BI Suite.
• There are over l30 active projects available for download on JasperForge. According to the forge statistics, 38 are public and 93 are private. Some of these projects use JasperSoft products, some extend them, and some provide embedding or integration of JasperSoft products into other applications. This led to a discussion of the Jasper4 program vs. other adapters; JasperSoft provides the functions in a Jasper4 branded application, such as Jasper4salesforce, while adapters or other applications from the community or third-parties would not carry the Jasper4 brand. CRM is a particularly active area as exemplified by the SugarCRM adapter, the partnership with Centric [see the OSA announcement] & the previously mentioned Jasper4salesforce.
• JasperSoft has 5000 payinq customers in 81 countries, approximately half are ISVs embedding JasperSoft capabilities into their own products, projects or offerings.
• JasperAnalysis & JasperServer are separate projects but share the same framework, which is why they were originally released on the JasperForge as JasperIntelligence, but they are currently being branded as Jasper BI suite; Ian gave a presentation showing the architecture and roadmap.

Overall it was an informative afternoon, and we're looking forward to working more with JasperSoft BI Suite in the future.

We're constantly recreating our 6D™ project management methodology. It started with combining Clarise's software development and project management experience with my aerospace system engineering and program management experience to adopt strict project controls to modern business needs for responsive software development and system integration processes working through distributed personnel. Well, here's a quick thought... software development and deployment should move away from traditional release cycle concepts to one of continuous process/code improvement within SaaS and virtual appliance environments. No code is alpha nor beta nor production, but a continuum of changes and adaptations responding to fluctuating business needs; done within a well managed environment to prevent security errors, poor performance, "garbage out" and junk code. So as we're assuring that our 6D™ [six dimensions of a project] is in accord with the PMBoK, we'll be keeping this thought in mind as well, and let's think beyond Extreme and Agile programming and continuous process improvement for software quality.

Our main interest in ESB is to supplement ETL, and inject real-time data into dashboards, OLAP, maybe even reports. BPEL/BPM as an offshoot of BI is also of interest. ServiceMix/Tuscany don't seem to be as useful in this regard. My guess is that ServiceMix/Tuscany will, as most Apache projects, be relegated to enhancing web servers, but not extend into other areas of the enterprise. Of course, everything might be pouring out of web servers in the future.

MuleCon was interesting mostly because of the enthusiasm of the customers. The week before we went to the Yahoo! sponsored open source content management system symposium [mostly to learn more about Alfresco, and to see the next generation of Joomla!]. That was pretty much all geek, all the time - very informative, but focused more on the technology, than the uses of the technology. MuleCon was smaller, and evenly distributed between presentations by customers and presentation by Mule core engineers. Not an hour went by without the phrase "use case" being bantered about. It was really very well done, despite some failure in conference organization. [It ended without a conclusion, just sort of petered out, with some folk in an unorganized discussion of the Scripps challenge, and others just milling about, wondering if the conference was over.]

People had come from all over the world, both to present and to participate; just over 100 were there for training on Tuesday, and about 150 were there on Wednesday. There were few Mule experienced people in the audience. Some had EAI or SOA/ESB experience [mostly Tibco] and were interested in Mule as an alternative, both for flexibility of adapting code, and for cost savings . Of course, the folk most willing to talk and to exchange business cards were other consultants. ;-) One interesting point in the side conversations was on the subject of configuration: some wanted wizards, others felt the XML files gave better control and facilitated understanding of the underlying process.

MuleSource seems very focused on their customers, and on providing business solutions through a flexible ESB without underlying infrastructure/technology assumptions. They should consider more such events, in varying locales. Hopefully, they learned as much about their potential users, as the participants learned about Mule.

We're blogged from MuleCon2007 throughout, and very much enjoyed the experience.