Everyone knows the old often-heard adage about the IT Department being more or less a neccessary evil – while this isn’t exactly true – many companies have operated that way over the past decade or so (many to their detriment, unbenowst to them).

Not only that, but the ever increasing convergence of tech and media has led companies to make technology purchasing decisions without involoving IT at all – only to roll it back on to the backs of already overburdened IT workers, sight-unseen.

There needs to be an organizational shift in the way companies view the traditional Business / Technology split among their own departments.

Recently, Warren Ritchie, CIO of Volkswagen Group of America was interviewed by ComputerWorld regarding the ever increasing use of technology in their present (and future) lines of cars. In discussing the data mining and interactive possibilites that would come along with “building cars … infused with intelligence, connectivity and new kinds of capabilities”, the conversation turned toward how this also posed an interesting problem for VWoA (or any shop looking to move forward in todays economy). “A lot of the services that we’re envisioning will require access to corporate functionality or corporate data”. Which breaks traditional IT/Biz paradigms regarding data and IT’s role in business or even customer support.

Basically, VWoA ended up revamping their IT structure by creating a kind of tech ambassador to ensure that Business and Information Technology flow together harmoniously toward a common goal.

…responding to these changes by creating a layer of people within his IT organization who are focused on understanding the business and its direction, and knowing how to architect future needs to emerging technologies…

Forward thinking. Make changes in the present to account for the future. Pro-active.

Ritchie and VW have created a “business technologist” position, someone who fills a consulting role to help people on the business side, make technology decisions.

A business technologist needs to have an IT background, coupled with an understanding of integration and the ability to speak the language of business.

Finally…

At C&C, we are glad to hear this type of news from such a high-profile source. We’ve been playing the role of “Business Technologists” for years now – and won’t stop anytime soon regardless of what the title is these days. The reality is that this type of “sweeper”, someone who can go between the business logic, requirements and bottom line of a company as well as the low-level “get your hands dirty” type of technology role is essential. If no one can effectively manage the progress and speak both languages things will get “lost in translation”, crippling its usefulness and possibly setting the stage for company-wide poor decisions based on bad information.

With the widespread adoption of Business Intelligence solutions keeping tabs on the actual heartbeat of the bottom line – any miscommunication can have severe financial reprocussions.

In a world where everyone is “moving to the cloud”, make sure that you have someone keeping their head OUT of the clouds keeping one foot in business and one foot in technology.

Front-office people in suits making gut-decisions about their business, and back-office IT workers in polo shirts primarily keeping the systems up just isn’t going to cut it anymore.

It is interesting how many organizations that we walk into struggle with reporting.  We will be implementing a large – scale application implementation with the database platform being SQL Server and inevitably someone will say, “Well, I guess we need to find some Cognos licenses, or go buy Crystal Reports or Business Objects.”  Not the case, if you have SQL Server, you have Reporting Services.

Sample conversation:

Client:  “Well, it’s called Microsoft SQL Reporting Services, I can only use the tool to create reports against SQL Server data, right Dave?  You mean to tell me that I have to automate moving my data from my Oracle and DB2 databases to use this reporting framework….too much work. Why exactly am I paying you again?”

Me:  “Let’s table that last question.   Well, while the Reporting Services Engine will use 2 databases that have to be on SQL Server you can develop reports directly from your Oracle or DB2 databases, or really any type of database platform that you have.  While in some cases it IS useful to move some of your data over to SQL Server, you really do not have to.”

Client:  “So how do people access these reports?  I probably have to run them myself or have one my people do it and then email them right?  That does not seem so Web 2.0?”

Me:  “Web 2.0…really?  Anyways, no, you will not run the reports, Reporting Services includes a reporting portal interface right out of the box.  Your folks open up their web browser and they have access to all the reports that are there based upon their permissions.  It’s really pretty slick.  Also, you can schedule reports to run on their own, keep snapshots of reports, and also have the system email the reports automatically to your clients.”

Client:  “You got an answer for everything, huh?”

Me:  “Yup.”

So the moral of that little diversion is that Reporting Services really is pretty powerful and most of you already have it already.

“Business Intelligence” is a term that’s been kicked around around for some time now, and basically just means “analyzing data from your past, in order to BETTER make better decisions in the future (or to better steer it in realtime)“. Its all about strategy, learning from mistakes (and successes), and being able to actively monitor / measure the health of your business. We might use this data to ask important business related questions: “Is Suzi Salesperson performing up to par this quarter?” or “What line of products have the biggest margin this time of year?”.

However, its not too far of a stretch to imagine someone who works for a Major League Baseball team sitting in front of a computer at the head office thinking: “What is the ROI of our pitching staff this year? Are they performing to expectations?” Its really no different then the business and reporting scenarios many of us encounter every single day. The exciting part is not only being able to read what has happened out on the ball field yesterday – but what might happen tomorrow. Boom. “Bizball Intelligence” anyone?

In this multi-part post, I’ll be taking you through all the steps needed to get up and running with your own historical AND current Major League Baseball statistics database (an operational / transactional DB), staging it out in a more “reportable” fashion (a data warehouse DB), and then finally building some cubes, calculated measures, and choice “Player Key Performance Indicators” (KPIs) based on the data. Who knows, maybe you’ll be so good at it that you’ll get hired by the San Diego Padres front-office, like this guy did, but more on him later.

So grab some sunflower seeds and a calculator – adjust your jock straps and get ready, ’cause here we go.

[ Part I: DATA ]

First up. We need some data (actually, a TON of data), and (more importantly) we need a place to store and easily access it. Its database creation time.

Here at C&C, we are big fans of the Microsoft SQL Server 2008 stack of products (Database, Analysis Services, & Reporting Services), and I’ll be using SQL Server 2008 R2 for this tutorial. You can get a 6 month eval copy, (you can also install the free SQL Server Express product to get up and running with a database at no cost – but you’ll be missing out on the Analysis Services and Data Mining pieces).

There are lots of places with baseball data on the web, but sometimes its incomplete, unofficial, not granular enough, and organized badly for what we need it for. Besides, we can’t do as much with stats that are ALREADY calculated – we need to calculate them ourselves in order to Data Mine properly (although those “other” data sources come in handy to verify that our own calculation formulas are correct). So in my mind, the grand-poobah of MLB data collection is (shockingly enough) MLB.com.

MLB.com’s “GameDay” application uses a XML structure to read info about every game, inning, pitch, hit and player. The “full-monty” of data only goes back to about 2006 or so (pitch location, coordinates, speeds, etc also known as “Pitch/FX Data”), but less granular data is available for prior years (and even more from other methods, I’ll post about later).

Browse around their backend, and you’ll see what I mean: http://gd2.mlb.com/components/game/mlb/. From there it goes into year/ month/ day/ game/ inning, linescore, batters, pitchers, etc. etc. all packaged up into XML files for the reading. Now before you think this is some kind of illegal backdoor in MLB’s servers, its not – well, not really. MLB has been aware that people have been using this data for about 4 years now. If they REALLY wanted to lock people out of it, they would have done it long ago. So feel free to browse it 100% guilt-free. The data is near realtime too! Pretty slick, MLB.

linescore.xml Example

So how do we get this mountain of XML files into our database?

Well, that’s easy. In 2010 Wells Oliver created a python script and associated libraries that reads the XML Gameday data and inserts it into a relational database format. Wells wrote the script to work with MySQL and (also) hasn’t updated it in some time (the MLB data format is always changing, and expanding, especially in 2011). So I took his fine work, rewrote it to use Microsoft SQL Server and added / changed / enhanced portions of its logic and the database schema. Anyways, I have “forked” his project, as they say in software development circles.

It requires:

• Python 2.6 (probably works with any 2.X ABOVE 2.6, but haven’t tried yet)
• Pymssql Python Library
• BeautifulSoup Python Library

Download it, expand it – (I use “C:\Gameday” myself), and then modify the ‘db.ini’ file in the root folder to configure your database instance.

I’m working on a development SQL instance, so my db.ini looks like this:

(If you need to use Windows Authentication / Trusted Auth, stay tuned because I’ll be adding that shortly)

Obviously you’re going to want to create a ‘mlbgameday’ (or whatever you desire to call it) database on the server first AND you’re going to have to create the schema frist.

Once you’re all set, the script is used like this: (make sure that the python directory is in your PATH)

The only required argument is “year”, but I would shy away from trying to fetch a whole year at one time for now. The multiple threads tend to overwhelm the pymssql library and max out some internal max connections setting (the TDS lib MAX_CONNECTIONS). For “type” you can fetch ‘mlb’ data or ‘aaa’ data, which might be useful in our analysis, but not a neccessity.

In order to gather a couple years worth of data painlessly, there is ANOTHER script I put together in the package called build_all.py – simply open this file with a text editor and modify these variables near the top:

Now, running “python build_all.py” will crawl through each date in your specified date range gathering data from all games on each day. Crawling through 6 years of MLB data might take a day or so – but once you have it, you have it – and updating it each day to get the previous days games should only take a minute or so (more on that in the next post).

Sample Output:

Etc… etc…

Fire it up and you’ll have more data then you know what to do with – what DO we do with it? That’s next…

The benefits of a connected mobile workforce are many, but the cost and effort of developing custom mobile solutions has precluded many organizations from making the move. Formotus offers a breakthrough solution that can connect your workforce directly to your business data sooner and with less up-front investment than ever before.

The Formotus Software + Services platform allows businesses to rapidly create and easily deploy powerful custom applications for mobile phones to connect field workers with real-time business data. The per-user subscription model lets you scale up your project as quickly or gradually as you want at the lowest possible level of cost and risk. You can also leverage your existing investment in the Microsoft Office system by using InfoPath as a rapid application development tool and SharePoint as an out-of-the-box solution for collaboration between your back office and your mobile workforce. The robust standards-based XML technologies at the heart of the Formotus solution make it compatible with any business systems that can be exposed as Web services. You can connect mobile workers with your own backend servers, whether running SAP, Oracle, SQL or Dynamics. Or you can connect to cloud-based solutions such as Google AppEngine, Amazon WS, or Microsoft SDS. Formotus is unique in its ability to rapidly create, customize and deploy rich data applications on Windows phones for the mobile workforce in any industry. General-purpose applications as well as industry-specific applications can be deployed swiftly using the powerful platform of Formotus Software + Services.

Formotus is the gateway to custom mobile business applications that are fast and affordable, with enterprise-grade reliability and security. Formotus solutions are device-independent and can be used by companies of any size.

Solution Description
We can help you create their own custom business forms without writing any code and run the same form on all kinds of devices including iPad, iPhone, Galaxy Tab, Xoom, and even ruggedized Windows Mobile 6.x devices. Not browser-based, these robust native apps are optimized to work well offline. This hosted mobile business application platform from Formotus works seamlessly with the Verizon Wireless network.

Top Reasons to Buy

• Go paperless. The easiest and most cost effective way to convert your mobile work force from paper to electronic forms.
• Go mobile. Use SharePoint or XML Web services to extend your office systems workflow out to your people in the field.
• Go fast. Forget app development, Formotus solutions are rapid to create, easy to manage, and work across many platforms.

For more information contact us and we will be happy to help mobilize your workforce.

• CMS092A – New York State OGS IT Services Backdrop Contract
• C000110 – New York State OFT Project Management MASA
• C000123 – New York State OFT Microsoft Support Services MASA
• HSN Agencies – New York State OTDA ITTPS Backdrop Contract

• SharePoint 2007
• SharePoint 2010
• Infopath Forms Server / SharePoint Integration
• ASP.NET, WPF, Silverlight, Windows Mobile Enterprise Application Development
• SQL 2005 / 2008 Database and Business Intelligence Platform (Reporting Services/Analysis Services / Integration Services
• Microsoft Dynamics CRM
• Exchange 2007
• System Center
• MS Office Communications Server