Absolutely right. I was lucky enough to learn this in college. Although, I did not learn it from the CS professors but rather my physics prof. He was a champion for a language called APL and he actually cut a deal with the CS department to accept credits for taking an APL class he was teaching as a substitute for the FORTRAN class. APL was an amazing mind-opening experience.
Throughout the APL 101 and 102 courses he would repeat this mantra: "Work on your data representation first. Once you have fully understood how to represent your data start thinking about how to manage it with code."
He would throw this at us permanently. At the time it sounded like our Physics prof had lost his marbles (he was a very, shall we say, eccentric guy). It would take a few years after college for me to realize the value of that advise.
Put another way, our business is managing data of some sort. Whether you work on embedded systems or web applications, you are always dealing with data. You can make your programs far more complicated than necessary by neglecting to choose the right (or a good) representation of your problem space (data).
I equate it to designing an assembly line. Anyone who's watched a show like "How it's Made" cannot escape the realization that efficient manufacturing requires engineering an efficient assembly process. Sometimes far more engineering work goes into the design of the manufacturing process and equipment than the part that is actually being made. The end result is that the plant run efficiently and with fewer defects than alternative methods.
In programming, data representation can make the difference between a quality, stable, bug-free and easy to maintain application and an absolute mess that is hard to program, maintain and extend.
Your prof was onto something that seems to be very in the zeitgeist today. To "understood how to represent your data" you have to understand what it is you're trying to represent. Eric Evans popularized this notion with Domain-Driven Design.
If you follow this line of thinking far enough, you realize that computer programming is just applied analytic philosophy. You have your metamodel (logic/programming language) and then you build your model (ontology/software).
I really like your assembly line metaphor. Knowing that a "customer" has a name and email address is almost of no importance compared to understanding how the "customer" information arrives, the actions around the "customer," and the end result of the actions. That's the assembly line.
I think that, to some degree, becoming an effective APL programmer almost required becoming good at data representation. If you want to do things the "APL way" you have to think about structures that work well with array, matrix and tensor processing ideas. When you could represent your data in the most optimal way you could sometimes write a function with only a few operators that could do what needed to be done to that data.
This means that, to some extent, data representation might have several local minima (in terms of being optimal) with each population of possible representations exists around the language or toolset you are going to use to process this data.
What I mean is that the optimal data representation for an embedded system (as an example) written in assembler might be different than that of the same system written in C or Forth.
I don't use APL very much any more. Back when I did I was also programming in Forth, C and Assembler. I know that the decisions I would make --in terms of optimal data representation-- varied from language to language. This is partly true because each language, in my case, was being used to deal with different problem domains. For example, APL could do high-level computations while Forth was great for low-level, real-time stuff.
Interesting comments. I've been working with an APL-derivative and this sounds similar to how I have been thinking about things. It's a matter of massaging the data to get in into a certain representation, e.g. a matrix, then from there it's very, very easy to work with using APL. As you say a few operators is all you need. The code is very terse. Very powerful. The real work seems to be shaping the data into the right representation first, a matrix. I use non-APL-descendent programs, the usual UNIX utilities, to do this for now.
Right. I remember having fun with this doing an APL application to aid in DNA sequencing. There were a number of ways to represent the data provided by the sequencer. At one point it almost became a game to see how small of an expression one could create to process the data by changing the representation.
With APL one has to be careful not to create monsters that cause geometric expansion in memory needs.
If the data set is large and the expressions processing the data cause frequent expansion into matrices or tensors (n-dimensional data structures where n > 2) one could end-up with geometric or exponential memory requirements. This, again, is another case of having to understand and fit data representation to the programming language AND the approach one will use to work with the data.
While languages like APL can be great, they can be disastrous in the hands of a programmer who does not understand what might be going on at a lower level. Sometimes there's nothing better than good-old low-level C.
Thank you, and you have just reminded me why I am liking the explosion of no-SQL stores - we have for a very long time been storing all our data in one factory design, with one, really flexible and powerful layout.
Being able to have a red black factory is rather nice. Although it does mean we now need to think carefully about what factory we shall need before even starting. And accepting occasionally moving the while factory three blocks over, during prodction
Throughout the APL 101 and 102 courses he would repeat this mantra: "Work on your data representation first. Once you have fully understood how to represent your data start thinking about how to manage it with code."
He would throw this at us permanently. At the time it sounded like our Physics prof had lost his marbles (he was a very, shall we say, eccentric guy). It would take a few years after college for me to realize the value of that advise.
Put another way, our business is managing data of some sort. Whether you work on embedded systems or web applications, you are always dealing with data. You can make your programs far more complicated than necessary by neglecting to choose the right (or a good) representation of your problem space (data).
I equate it to designing an assembly line. Anyone who's watched a show like "How it's Made" cannot escape the realization that efficient manufacturing requires engineering an efficient assembly process. Sometimes far more engineering work goes into the design of the manufacturing process and equipment than the part that is actually being made. The end result is that the plant run efficiently and with fewer defects than alternative methods.
In programming, data representation can make the difference between a quality, stable, bug-free and easy to maintain application and an absolute mess that is hard to program, maintain and extend.