The Importance of Data and Data Order

I received a refresher course on the importance of Data Order recently which also reminded me of a central truth about data that I came up with years ago. I thought it worthwhile to share the experience.

Here is the scenario: I have a cursor that I build from scratch inserting records from various tables during a long process. The last process in building the table is to "articulate" some credits and pushing the articulated credits to a 4th level node. The end result is a table that is then presented to the user in a dbi treeview control. The end result is a college degree plan with subject groups (requirements) and credit records which meet the subject group requirement.

Some colleges want to articulate transfer credits and other nontraditional credits into their own courses. That is what I am calling articulation.

In the image below is the end result in the treeview. The articulating courses are in bold with their respective articulated courses beneath them; articulated course rows also have a light green background.



On the form that has the treeview is a command button called Import New. When clicked, credit records that have been added since the degree plan was last saved are inserted into the underlying treeview cursor in the correct position. The image below shows the treeview after importing new but not articulating any imported new credits. Newly imported records have a forecolor of blue and it's icon has a yellow backcolor.



This has worked fine and the users have the ability to manually articulate credit records if they so desire within the treeview control.

Wouldn't it be great to automatically articulate credit records during the import new process. I thought so and so did the users; that means less manual articulation. Below are the initial results.



This is an absolute mess!!! There are credit records that are children of other credit records and the articulation records have lost their children. I think there might be a problem with the order. But why, this has always worked before. I didn't make any changes to the Import New or to the Articulation processes.

Here is the code I found at the end of the processing where I create an index on the cursor before filling the treeview control with the data.



I can see why the order got mixed up now. If plImportNew is TRUE, the index being built just doesn't cut it. Before I ever started doing articulation during processing this all made sense. When I stated doing articulation during processing I had to create a crazy looking index to get the order correct for display. I never messed with the index if plImportNew was TRUE because I didn't allow articulation during Import New.

So, I decided that in order to fix this I would take the IF statement out. I could use the same index all the time. Below is what I got for this simple and wonderful solution.



Not quite what I was hoping for. In some ways it is better; it's not as mixed up as it was and some records actually look correct, but there are still misplaced records and the old records that were correct now look misplaced. What is going on? My index looks correct.

That's when it hit me. Look at the data! I have this saying, "The data never lies". In order to get to the heart of the matter, look to the data and it will show you the truth. So I did, and the data showed me what the problem was. I have a field called nParent which holds the recno() of the parent node. This value held the correct value upon the last save. But after importing new records and inserting them into the cursor the parent's recno changed. The child's nParent value held the old value, not the new value. So, when the index was created to set the correct display order the records ended up out of order because of the wrong values.

In order to correct this problem I had to replace the nParent value of the child records after the Import New process but before the Articulation process. This is only done for records that are "articulated" records and are 4th level nodes in the treeview. The code is shown below.



Now the results look as expected and are shown below.



The data is now in the order that it should be. The index is working correctly. In order to get there, I had to look at the data. The data tells the truth. Now I am very close to having a program that is ready for release. We all know the importance of the order of data; it can give new meaning to the displayed results just by changing the order of the records. When the displayed results are not quite what you had hoped for, try looking at the raw data to see if the answer lies there.

Note:
Before I start getting messages about how I shouldn't use recno() for field values or for indexing let me say this is an exception. I have created a cursor to hold data. The data is actually stored in a memo field when the degree plan is saved. The user has the ability to drag and drop nodes all over the place within the treeview control. I didn't want to have to mess with record order in a normal table. By saving to the memo field row by row the order in the treeview is preserved and it is easy to reload the treeview form the memo field.

FoxTales Review

This is a very short review of FoxTales by Kerry Nietz. I received this book free at SWFox2010. I enjoyed reading this book. It is written by one of the programmers of FoxPro and covers a little history of Fox Software from the FoxBase days to the company being sold to Microsoft.

After being sold to Microsoft there is but one single chapter which is more of a summery of where the different programmers went to within the Microsoft organization. I would have liked to have read about the Microsoft days too. Maybe Kerry can be persuaded to write a 2nd book covering that time period.

Reading this book was a walk down memory lane for me. I used FoxBase+. Before I purchased FoxBase I was using dBase III+. I had written a bicycle race program that I was using at a bicycle race. Well, my program froze on me and it was an anomaly in dBase III +. I immediately bought FoxBase+ and my problem went away. Later I moved up to FoxPro 2.6 for DOS. I had bought the Windows product too by I didn't like it. I had even used the Mac version of FoxPro, but it didn't do much for me either. I used FoxPro 2.6 for DOS for a long time. When I found Visual Fox Express, I moved up to VFP 6.0 where I have been for a long time. Enough of memory lane.

I dogeared a page at the end of Chapter 15. The passage that caught my interest was from the 2nd Fox Devcon at the "FoxPro Feedback " session at the end of the conference. A woman from the audience spoke:

"I'd just like to take this time to thank those of you on the panel. You and your product really changed my life. Before Foxro, I was living in a tent on the street, unsure of where I'd get my next meal. But now I'm a highly skilled and well-paid consultant... and I have you guys to thank."

This made Kerry feel very appreciated and he says "Sometimes the blessings take time to see." referring to all the work he had done himself on FoxPro and feeling under-appreciated up to that point. Kerry then understood the importance of his work and the FoxPro product. People were making a good living using the product that he helped build. This section made me think a lot about developers on every level.

I use a couple of frameworks. I have tried to make a point to the developers of those frameworks how important their work is too me. I use these frameworks for the products that I've created. I honestly believe that I could have never created these programs without the frameworks that I use. I have been able to make a good living because of FoxPro and the frameworks that I use. I am deeply indebted to the developers for FoxPro, Visual Fox Express, and Web Connection.

I am also very proud of the fact that there are many users of my programs. My programs have actually created good paying jobs. Not only has my program created jobs for those who use the program, but the output of my programs helps people as well. My programs have touched thousands of people. Hopefully their lives are better because of the work that I have done; I certainly hope so. It is that knowledge of helping others that at times keeps me going.

I think we as software developers sometimes forget the impact that we have made and continue to make. Sometimes we even feel like we have been taken for-granted. For the most part that's ok with me. I consider myself a humble person and I've never wanted to be in the spotlight or famous. But I do want to do good in this world and I do want to make a positive difference.

I was reading Keven Ragsdale's blog not long after attending SWFox2010. In one of his post he talks about being self-taught. I've used that line too. I taught myself FoxPro as has all the FoxPro programmers that use to work with. Kevin says NOT! Kevin says we are deeply indebted to the writers of the publications and books that we have read over the years for teaching us. So true. Read his blog post, a three part series, starting here. These are older posts, but I'm sure you will find that what Kevin says applies to you as well. I know that it does for me.

So, I encourage all of you to acknowledge the developers that have helped you realize success. And if you are one of those developers who has created a framework or other tool please know that you are appreciated and because of you and your work the community has been blessed in immeasurable ways. I also encourage every one to acknowledge the writers that have given so much.

Help Builder and Snagit

Last week I spent time getting re-familiar with Help Builder by Rick Strahl. I had created a help file years ago which is used by my primary application. I really liked using Help Builder back then, but I never updated my help file. So I decided to start updating the help file.

First I made sure I had the latest version of Help Builder. Rick has made some enhancements that I believe the programmer will like for building program documentation (that's another thing I haven't done either).

Next I purchased and downloaded Snagit v10. In the past I had used Paint Shop Pro's (PSP) capture software. Using PSP worked but it was also kind of a hassle; I think that is why I let the help file get so far behind. Well, Rick has done something special. Help Builder will interface with Snagit. This process works really good and makes capturing screen shots a breeze. There is no excuse to not have plenty of screen shots in the help file now.

I have to give Rick a big thanks for another wonderful product.

I've never seen my users use the help file. Maybe because it is so old, or maybe because they just don't know about it. Recently I've been reading the help files from different frameworks that I use and there is a wealth of information in there. It is my hope that I can create a help file that is just as useful as the ones I've been using lately. I hope to have help project completed soon and I have Help Builder and Snagit to thank.

Resizer Control

I've been playing with the Resizer Control in VFE the last few days. It has been a lot of fun and the guys at the office like it. I can't believe I've never used it before.

In the past I've made my screens to lower resolution monitors as that is what some of my user's use. With the Resizer the users can make a form what ever size they want.

Now if I can just figure out how to save the changed size settings so that the user doesn't have to do this every time they open a form. I'll have to come back to this later as I have other things that need my attention. The Resizer was a fun diversion.

SWFox 2010 Review

This was my 2nd year in a row of attending a SWFox conference. Each conference I attend is just as rewarding as the last.

I arrived early this year to attend Rick Strahl's Double Impact preconference session, a two day event. As I am still working in v2.97 of Web Connection I really wanted to see what v5.53 looked like. All I can say is wow!. Rick is a great presenter and developer. He showed us some cool stuff. The things I brought back with me: AJAX, jQuerry, and the need to learn CSS syntax. That might not seem like much but there is plenty of documentation for me to read through to get the details. Rick writes some of the best documentation plus he is active on his message board to answer questions if a person can't find the answers in the documentation .

Thursday night was the opening session with a keynote by Craig Boyd. Craig delivered the best keynote I have ever heard. To start was the entertainment value, very entertaining and captivating. Then came the serious part of the keynote. Craig talked about perceived value; this is the value the customer perceives in the products developers deliver. The mirror image is the value that we as developers perceive in using VFP, but more importantly it is the skill that we as developers use to find solutions to customer's problems. It really shouldn't matter what tool (VFP) we use as long as we find the best solution. Craig also emphasized that VFP9 is based upon C++ v7.1 as a codebase. He does not see Microsoft developing a OS that would cause programs written with the 7.1 codebase to no longer work for a long, long time.

Craig's keynote was very inspirational to me; I really needed to hear his message. For me it validated what I do and confirmed what I have been thinking. I love FoxPro. I love creating solutions using FoxPro. I believe that I am good at what I do. I want to continue to use FoxPro to create solutions for customers. But... I acknowledge that I also need to learn new tools. For me, the inspiration and motivation I received from Craig's keynote was worth the costs of attending the conference. Thanks Craig.

When I go to a conference there are certain speakers I will not miss. My first session on Friday was Toni Feltman's The Pomodoro Technique. Since returning home from the conference this is the one session I have thought about the most. Time management. I'm not very good at it; I hope to apply the technique Toni presented so that I can be more productive. If I'm more productive I will also feel better about myself. This technique is about commitment and self-discipline, but in small pieces. I can do that.

Another speaker I really enjoy is Doug Hennig. I attended both of his sessions (Themed Controls and Cool Controls). Doug exhibits so much energy during his presentations. The 75 minutes fly and before you know it the session is over. I saw some really cool stuff; now I just have to figure out how and where to use it.

The third speaker I didn't want to miss was Rich Schummer. I attended his session on Mocking Your Customer. Rick explained that he uses mockups during design instead of creating sample applications. The benefits are: 1) It takes less time. 2) The customer understands that it is a mockup, not a running application. Rick explained that when a customer sees a running demo that the customer has the perception that the project is almost complete; whereas with a mockup the customer understands that the coding hasn't even begun. A great point in my opinion.

Friday had two Show Us Your App sessions. One before dinner, and one after. During each session I saw at least one cool application. I have to say that the FoxyPreviewer was really cool.

Saturday was a full day of sessions for me. I attended both of Jody Meyer's sessions (Web Development Using CSS and PDF Output). While these sessions were aimed more at beginners for these topics, I learned something from both sessions. Jody's organization and presentations were superb. An observation I made at both sessions were the amount of questions. It was clear that there was a lot of interest in these topics and it was something new for many of the attendees.

I only went to two session on Sunday, both of which hit the WOW! button for me. The first was Cathy Pountney's fxReports. She demonstrated customizing the report writer to add a tab for watermarks. I thought this was really cool. I sure wish I had attended her session last year on the subject. The 2nd session was Steve Ellenoff's Integrating Windows 7 Taskbar into VFP. He showed some really cool things you can do with the Windows 7 taskbar using WinyTLib which is posted on VFPX.

There are a few sessions that I wish I had attended, but at least I have access to the white papers. There are a couple of pre-conference sessions I would have liked to have attended, but I was committed to the Double Impact sessions; I would have really liked to have seen Ruby on Rails. Also, I would have liked to have stayed an extra day to see all the sessions on Silverlight. That being said, I was ready to come home after a week in AZ.

I probably spent as much time socializing as I did attending sessions. It was really interesting to be around all these developers from around the world and the conversations that would take place. I made at least one new friend. There is more to going to a conference than just attending sessions. I think it was time well spent, plus it was quite enjoyable.

Something new this year at SWFox was online session evaluations. From what I can tell it was a success. With the paper evaluations I tended to quickly fill them out at the end of each session. With the online evaluations I could make the evaluations when I wanted and I wasn't restricted as to the space on the paper to make comments. I liked the online evaluations much better than the paper version.

There was discussion about evaluations that I believe is worth mentioning. Currently ratings are on a 5 point scale. There was a suggestion that ratings be on a 10 point scale. For example: a session wasn't quite a 5 but more than a 4; on a 10 point scale that would be a 9. Personally I'm for the 10 point scale. While on the subject of evaluations, I had a discussion with another attendee about where does one start as a basis when making evaluations. I tend to start with 4 as all sessions are better than what I would consider average. My friend starts with a 3. I found this very interesting and makes me wonder where others start as a basis when making evaluations.

Other trends that the organizers are setting is going green, at least that's what I'll call it. First, all conference materials are given to attendees at registration are in a re-usable grocery bag. I like this; I re-use the bags all the time. It sure beats those satchels I've received at other conference that I never use again. Another trend is not having the notebook binders with all the paper inside. We all received CDs with the white papers and examples that each presenter created; all excellent material. Who needs all those notebooks and all that paper. And, who wants all that extra weight to carry back home? I have enough trouble keeping my bag under 50 lbs when you consider I brought home some extra books too.

The organizers of SWFox do an excellent job of putting on conferences. They really put their hearts and souls into this and we as the VFP community reap tremendous rewards. I am truly thankful for their effort and the efforts of the speakers. I have come back from this conference refreshed and motivated, but I have also come back with a different perspective (being involved). We as a community need to help the organizers where we can. There is the obvious; attend the conference and give feedback. They read the feedback and take it to heart. Another way to participate is to make suggestions on how to make the conference better and what sessions to present. I have been trying to come up with session topics and relaying these to the organizers. I encourage everyone to suggest a topic for next year's conference.

I hope to see everyone at next year's SWFox. My plans are to attend. See you there.

First Post

My intention is to blog about my professional endeavers of software development. My current development tools are: Visual FoxPro, Visual Fox Express, and Web Connection.