NYCPHP Meetup

[Hans C. Kaspersetz]

> - one problem with the "Build System B and migrate to it" approach is 
> that you might build system B from scratch and think that it's really 
> great,  but discover that the migration isn't really possible once you 
> try it.
>
+1.........  You never really know the gremlins that live deep in that 
database that make ETL or migration a complete fiasco until you run your 
migration scripts several hundred times trying to get all your data to 
look right to the end users. If it was a custom app, you never know when 
some programmer, or several personalities, a couple of years ago decided 
to change the date format and has included a patch in his code to deal 
with the two or more formats but didn't tell you.  You go to key off 
that date and you find your scripts inexplicably failing.  You look 
deeper and find 6 different date formats in the same column.  Who would 
have guessed we store dates in 6 different ways?  Hmm.... then you 
discover that not all accounts have all information because columns were 
added as time went on and requirements grew. 

It seems so easy to move data from flat files (access) to a relational 
database in concept.  In a trivial system it might be easy, however the 
cost associated do not grow linearly.  It seems to grow much faster then 
the perceived complexity of the problem, ask Airbus.  They just had some 
wires that were the wrong length.

Then you start to try to take advantage of RDMS and you find that 
incomplete data or data that is not keyed properly doesn't move 
gracefully.  You find that the original design of the flat files falls 
short and needs to changed to support the new app.  So you start to 
transform the data to fit the new schema and you find that keys you 
think would be no brainers don't exist or are incomplete.

On top of that, you design all these new data structures and classes to 
do really neat things.  Your very proud of yourself and the way the 
application behaves with test data.  Then you shoe horn the old data in 
and your application looks more like a cubist painting then a crisp 
photo from a Canon 20D with an L lens.  Then you spend some more time 
massaging the data.  In the end it is beautiful, but it requires a high 
level of expertise and the constitution of Conan.

I speak from direct personal experience working with data collected over 
the course of 20 years.  It is not impossible, I know, we are 
successfully doing it here.  I happen to be blessed with working with a 
very talented team of people.  But you have to be very very aware that 
there are things that you don't know that you don't know. Ha ha ha....  
It is a very painful lesson to stand in front of the CEO and CFO and 
explain that you just spent 160 man hours working on how to retrieve and 
correctly key broken data from the old system.  Data everyone thought 
was locked down until someone in sales says they can't find some over 
looked piece of data when he is testing.  Your story may not be the 
same, but your feelings will be.

Hans Kaspersetz
NJ Web Development by http://www.cyberxdesigns.com